Classic Models

Headlines are about giant neural networks, but a great deal of production machine learning — credit scoring, demand forecasting, fraud flags, churn prediction — runs on models invented decades ago. They train in seconds on a laptop, need hundreds of examples rather than millions, and many of them can explain their decisions. On tables of numbers and categories, the kind of data businesses mostly have, they are often as good as anything newer. Knowing them well is not nostalgia; it is how you pick the right tool.

One interface, many models

Everything here uses the workflow from The Machine Learning Workflow: split the data, fit on the training part, score on the test part, and compare with a baseline. scikit-learn gives every model the same methods — fit, predict, score — so swapping one model for another is a one-line change. That consistency is why the library became the standard for classic machine learning.

Examples on this page were run with Python 3.13, scikit-learn 1.9 and pandas 3.1 As of 2026, with fixed random seeds.

The six models fall into three families, depending on how they turn features into a prediction:

• Weighted sums — linear and logistic regression multiply each feature by a learned weight and add them up.
• Questions — decision trees and random forests ask a sequence of yes/no questions about the features.
• Distances — k-nearest neighbours and k-means decide by which examples or centres are closest.

Each family has a characteristic strength and blind spot, and knowing which family a model belongs to tells you most of what you need: whether it needs scaled features, whether it can capture “if this and that” interactions, and whether you can explain its decisions.

Linear and logistic regression

Linear regression predicts a number as a weighted sum: prediction = w₁ × feature₁ + w₂ × feature₂ + … + a constant. Training finds the weights that make the squared errors on the training data as small as possible — the “least squares” method published by Adrien-Marie Legendre in 1805 and Carl Friedrich Gauss in 1809. It is the oldest model on this page and still one of the most used.

from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

X, y = load_diabetes(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

model = LinearRegression().fit(X_train, y_train)
print("R^2 on test set:", round(model.score(X_test, y_test), 3))
for name, coef in zip(X.columns[:4], model.coef_[:4]):
    print(f"{name:>4}: {coef:8.1f}")
print("Prediction for first test patient:", round(model.predict(X_test[:1])[0], 1),
      "| actual:", y_test.iloc[0])

R^2 on test set: 0.485
 age:     47.7
 sex:   -242.0
 bmi:    532.0
  bp:    381.6
Prediction for first test patient: 137.9 | actual: 219.0

The diabetes dataset has ten measurements for 442 patients and a score of disease progression one year later. The R² of 0.485 says the model explains roughly half of the variation in that score — useful, far from perfect. Its great virtue is that you can read it: each coefficient says how much the prediction moves when that feature goes up. In this dataset the features have already been centred and scaled, so the sizes are roughly comparable, and body-mass index (bmi) and blood pressure (bp) push the prediction up most. Be careful, though: a coefficient describes the pattern in this data, not a cause, and the negative weight on sex just reflects how that column happens to be coded.

Logistic regression, despite the name, is for classification. It computes the same kind of weighted sum, then squashes it through an S-shaped curve into a probability between 0 and 1.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42
)

model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)
print("Test accuracy:", round(model.score(X_test, y_test), 3))
probs = model.predict_proba(X_test[:4])[:, 1]      # chance of label 1 (benign)
print("P(benign) for four patients:", probs.round(3))

Test accuracy: 0.986
P(benign) for four patients: [0.969 0.    0.559 0.939]

98.6% on the tumour data, as on the previous page. The useful part is predict_proba: the model is very sure about the first, second and fourth patients (the second gets a probability of benign that rounds to 0 — confidently malignant) but only 56% sure about the third. In practice that third case is the one to send to a human. Linear models are fast, stable with little data and easy to explain, which is why regulated industries such as banking favour them. Their limit is in the name: they assume each feature pushes the answer steadily in one direction, so they miss curves and interactions (“high blood pressure matters only above a certain age”) unless you build those features yourself.

Decision trees and random forests

A decision tree learns a flowchart of yes/no questions. At each step it picks the feature and threshold that best separate the classes, then repeats on each side. The modern version was set out in the 1984 book Classification and Regression Trees by Leo Breiman, Jerome Friedman, Richard Olshen and Charles Stone. You can print a small tree and read it.

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier, export_text

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.3, stratify=iris.target, random_state=42
)

tree = DecisionTreeClassifier(max_depth=2, random_state=42).fit(X_train, y_train)
print(export_text(tree, feature_names=list(iris.feature_names)))
print("Test accuracy:", round(tree.score(X_test, y_test), 3))
print("Classes:", iris.target_names.tolist())

|--- petal length (cm) <= 2.45
|   |--- class: 0
|--- petal length (cm) >  2.45
|   |--- petal width (cm) <= 1.55
|   |   |--- class: 1
|   |--- petal width (cm) >  1.55
|   |   |--- class: 2

Test accuracy: 0.889
Classes: ['setosa', 'versicolor', 'virginica']

The iris dataset measures 150 flowers of three species. With only two questions, the tree has found a rule a botanist could check by hand: petal length 2.45 cm or less means class 0 (setosa); otherwise, petal width decides between versicolor and virginica. 88.9% accuracy from a rule that fits on a sticky note. max_depth=2 is what kept it small; on the previous page, an unlimited tree scored a perfect 1.0 on its training data and overfitted. Trees need no feature scaling, handle mixed kinds of data and capture interactions naturally — but a single tree is unstable: change a few training rows and it may ask entirely different questions.

A random forest, introduced by Leo Breiman in 2001, fixes that instability with numbers. It grows hundreds of trees, each on a random sample of the rows and considering a random subset of features at each split, and lets them vote. Individual trees make different mistakes; the vote cancels many of them out.

import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42
)

forest = RandomForestClassifier(n_estimators=200, random_state=42).fit(X_train, y_train)
print("Test accuracy:", round(forest.score(X_test, y_test), 3))
importance = pd.Series(forest.feature_importances_, index=X.columns)
print(importance.sort_values(ascending=False).head(4).round(3))

Test accuracy: 0.958
worst perimeter         0.146
worst area              0.137
worst concave points    0.107
mean concave points     0.104
dtype: float64

On the same split where one unlimited tree scored 92.3%, two hundred trees score 95.8%. The forest also reports feature importances — how much each feature helped across all its splits — here dominated by size and shape measures of the cell nuclei. Treat these as hints rather than proof: importances can be shared or split between correlated features. Random forests are perhaps the best “first serious model” for tabular data: robust, hard to break with default settings, and rarely embarrassing. Their cousins, gradient-boosted trees (in libraries such as XGBoost and LightGBM, and scikit-learn’s own HistGradientBoostingClassifier), are frequently the winners on tabular problems. The price is readability: you can print one tree, but not a forest of 200.

Nearest neighbours: similar things, similar answers

k-nearest neighbours (k-NN) barely trains at all. It stores the training data, and when asked about a new example it finds the k most similar stored examples and takes a vote. It goes back to work by Evelyn Fix and Joseph Hodges in 1951. “Similar” means close in distance, which makes the scale of the features decisive.

from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_wine(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, stratify=y, random_state=42
)

raw = KNeighborsClassifier(n_neighbors=5).fit(X_train, y_train)
scaled = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5)).fit(X_train, y_train)
print("k-NN, raw features:   ", round(raw.score(X_test, y_test), 3))
print("k-NN, scaled features:", round(scaled.score(X_test, y_test), 3))

k-NN, raw features:    0.722
k-NN, scaled features: 0.944

The wine dataset has 13 chemical measurements of 178 wines from three cultivars. Without scaling, k-NN gets 72.2%; with feature scaling, 94.4%. Why? One feature, proline, runs into the hundreds and above a thousand, while others are fractions. Unscaled, the distance between two wines is almost entirely their difference in proline, and the other twelve measurements are ignored. Scaling gives every feature a fair say. k-NN is simple to understand and explain (“these five past cases looked most like yours”), but it slows down as data grows because every prediction compares against every stored example, and it struggles when there are many irrelevant features. The same nearest-neighbour idea, on embeddings, powers semantic search later in this track.

k-means: finding groups without labels

Every model so far needed labels. Clustering works without them: it groups examples so that similar ones end up together — a form of unsupervised learning. The classic algorithm is k-means: place k centre points, assign every example to its nearest centre, move each centre to the average of its group, and repeat until nothing changes. (Stuart Lloyd described the method at Bell Labs in 1957, published in 1982; James MacQueen named it “k-means” in 1967.)

import pandas as pd
from sklearn.cluster import KMeans
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler

iris = load_iris()
X = StandardScaler().fit_transform(iris.data)   # no labels used from here on

kmeans = KMeans(n_clusters=3, n_init=10, random_state=42)
clusters = kmeans.fit_predict(X)

# only now peek at the real species, to see what the clusters found
species = pd.Series(iris.target_names[iris.target], name="species")
print(pd.crosstab(species, pd.Series(clusters, name="cluster")))

cluster      0   1   2
species               
setosa       0  50   0
versicolor  39   0  11
virginica   14   0  36

We hid the species from k-means and asked for three groups. Only afterwards did we compare the clusters with the real species. Setosa came out perfectly as its own cluster (cluster 1 — the numbers are arbitrary names). Versicolor and virginica, which overlap in size, were split imperfectly: 25 of the 100 landed in the “wrong” group. That is typical: k-means finds structure in the measurements, which may or may not match the categories people care about. It also needs you to choose k, assumes roughly round, similar-sized clusters, and is sensitive to scaling — hence the StandardScaler. Use it for exploration (“what kinds of customers do we have?”), and always ask a person who knows the domain whether the groups make sense.

Choosing a model

Here are five of the models side by side on the wine data, scored with five-fold cross-validation, with the baseline first.

from sklearn.datasets import load_wine
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier

X, y = load_wine(return_X_y=True)
models = {
    "Baseline (most frequent)": DummyClassifier(strategy="most_frequent"),
    "Logistic regression": make_pipeline(StandardScaler(), LogisticRegression()),
    "Decision tree": DecisionTreeClassifier(random_state=42),
    "k-nearest neighbours": make_pipeline(StandardScaler(), KNeighborsClassifier()),
    "Random forest": RandomForestClassifier(random_state=42),
}
for name, model in models.items():
    scores = cross_val_score(model, X, y, cv=5)
    print(f"{name:25} {scores.mean():.3f}")

Baseline (most frequent)  0.399
Logistic regression       0.983
Decision tree             0.865
k-nearest neighbours      0.949
Random forest             0.972

Every real model beats the 39.9% baseline by a wide margin. The scaled logistic regression wins here, ahead of the random forest — a reminder that the simplest model is sometimes the best, and that you only learn which by trying. On a different dataset the order would change. A fair comparison like this loop is how to choose.

Key terms on this page

Sources & further reading

• scikit-learn developers, Supervised learning and Clustering (scikit-learn user guide).
• scikit-learn developers, Decision Trees, including export_text (scikit-learn user guide).
• Leo Breiman, Jerome Friedman, Richard Olshen & Charles Stone, Classification and Regression Trees (Wadsworth, 1984).
• Leo Breiman, “Random Forests”, Machine Learning 45 (2001).
• Stuart P. Lloyd, “Least squares quantization in PCM”, IEEE Transactions on Information Theory 28(2) (1982).
• Léo Grinsztajn, Edouard Oyallon & Gaël Varoquaux, “Why do tree-based models still outperform deep learning on tabular data?”, NeurIPS Datasets and Benchmarks (2022).
• Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani & Jonathan Taylor, An Introduction to Statistical Learning, with Applications in Python (Springer, 2023; free online).