Measuring Models

A colleague proudly reports that her new model is 90% accurate at spotting the digit nine in handwritten numbers. It sounds respectable — until you discover that the model never says “nine” at all. It answers “not a nine” to every single image, and because only one image in ten is a nine, it is right 90% of the time. Nothing was learned; nothing is useful. This page is about not being fooled like that — by other people’s numbers, and above all by your own.

Why accuracy misleads

Accuracy is the share of predictions a model gets right. It is the first number everyone reaches for, and for balanced problems it is fine. But many real problems are lopsided. Fraudulent card payments are a tiny fraction of all payments; most screening tests come back negative; most emails in a well-filtered inbox are not spam. When one class dominates — class imbalance — a model can score high accuracy simply by always predicting the common class.

Here is the opening story as code. We use scikit-learn’s bundled handwritten-digits data, turn it into a yes/no question (“is this a nine?”), and ask a deliberately lazy model, DummyClassifier, to always predict the most frequent answer. If you have not met train_test_split before, the machine learning workflow page explains it.

from sklearn.datasets import load_digits
from sklearn.dummy import DummyClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, recall_score

X, digit = load_digits(return_X_y=True)
y = (digit == 9)                      # True only for images of a nine
print("Share of nines:", round(y.mean(), 3))

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

lazy = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
pred = lazy.predict(X_test)
print("Accuracy:", round(accuracy_score(y_test, pred), 3))
print("Nines found (recall):", recall_score(y_test, pred))

Share of nines: 0.1
Accuracy: 0.9
Nines found (recall): 0.0

y = (digit == 9) turns the ten digit labels into True/False. stratify=y makes the training and test sets keep the same one-in-ten share of nines, which matters for imbalanced data — otherwise a small test set might, by bad luck, contain almost none. The lazy model scores 90% accuracy and finds zero nines. That last number, recall, is one of the measures this page will teach you to read alongside accuracy.

The confusion matrix

To see how a classifier is wrong, lay its predictions out in a two-by-two table called a confusion matrix. One axis is the truth, the other is the prediction. Every prediction lands in exactly one of four cells. Tap each tile; we will use a medical test as the running example, where “positive” means “the test says disease”.

• TPHITTrue positiveThe patient is ill and the test says ill. The thing you built the test to do.
• FPFALSE ALARMFalse positiveThe patient is healthy but the test says ill. Costs worry, follow-up tests and money. In statistics, a “type I error”.
• FNMISSFalse negativeThe patient is ill but the test says healthy. They go home untreated. Often the most dangerous cell — a “type II error”.
• TNALL CLEARTrue negativeHealthy, and the test agrees. In imbalanced problems this cell is huge, which is exactly why accuracy looks so good.

Now a real model. scikit-learn’s breast-cancer dataset has 569 tumour samples described by 30 measurements taken from cell images, each labelled malignant or benign. In this dataset the label 0 means malignant, so we flip it to make “malignant” our positive class. We train a logistic regression (from Classic Models) and print the confusion matrix and scikit-learn’s classification report.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import confusion_matrix, classification_report

data = load_breast_cancer()
y = (data.target == 0)                # positive = malignant (in this dataset 0 means malignant)
X_train, X_test, y_train, y_test = train_test_split(
    data.data, y, test_size=0.25, random_state=42, stratify=y)

model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
model.fit(X_train, y_train)
pred = model.predict(X_test)

print(confusion_matrix(y_test, pred))
print(classification_report(y_test, pred, target_names=["benign", "malignant"], digits=3))

[[89  1]
 [ 4 49]]
              precision    recall  f1-score   support

      benign      0.957     0.989     0.973        90
   malignant      0.980     0.925     0.951        53

    accuracy                          0.965       143
   macro avg      0.968     0.957     0.962       143
weighted avg      0.966     0.965     0.965       143

scikit-learn prints the matrix with true classes as rows and predictions as columns, in label order (False = benign first). So reading it: 89 benign tumours correctly called benign (TN), 1 benign tumour wrongly flagged (FP), 4 malignant tumours missed (FN) and 49 malignant tumours caught (TP). Accuracy is 96.5%, which sounds excellent — but four people with cancer would have been told they were fine. The matrix makes that visible; the single accuracy number hides it. make_pipeline(StandardScaler(), …) rescales each measurement before the regression sees it, which helps logistic regression train smoothly.

Precision, recall and F1

Two numbers summarise the positive class far better than accuracy.

• Precision = TP ÷ (TP + FP). When the model says “malignant”, how often is it right? Here 49 ÷ 50 = 0.980.
• Recall = TP ÷ (TP + FN). Of all the malignant tumours, how many did the model catch? Here 49 ÷ 53 = 0.925. Doctors call this sensitivity.
• F1 score = 2 × precision × recall ÷ (precision + recall), the harmonic mean of the two. It is high only when both are high, so a model cannot game it by maximising one and ignoring the other.

A spam filter makes the difference vivid. If it has high recall but low precision, it catches every spam message — and also throws your bank’s genuine statement and your cousin’s wedding invitation into the junk folder. If it has high precision but low recall, everything it junks really is spam, but plenty of spam still reaches your inbox. Which is worse depends on the cost of each mistake, which we will come back to.

Most classifiers do not really output “yes” or “no”; they output a probability, and predict simply applies a threshold of 0.5. Move the threshold and you trade precision against recall. This snippet uses the same model and data as before:

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import precision_score, recall_score, f1_score

data = load_breast_cancer()
y = (data.target == 0)
X_train, X_test, y_train, y_test = train_test_split(
    data.data, y, test_size=0.25, random_state=42, stratify=y)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_train, y_train)

p_malignant = model.predict_proba(X_test)[:, 1]   # probability of the positive class
for threshold in [0.9, 0.5, 0.1, 0.01]:
    pred = p_malignant >= threshold
    print(f"threshold {threshold:<4}  precision {precision_score(y_test, pred):.3f}"
          f"  recall {recall_score(y_test, pred):.3f}  F1 {f1_score(y_test, pred):.3f}"
          f"  flagged {pred.sum()}")

threshold 0.9   precision 1.000  recall 0.811  F1 0.896  flagged 43
threshold 0.5   precision 0.980  recall 0.925  F1 0.951  flagged 50
threshold 0.1   precision 0.897  recall 0.981  F1 0.937  flagged 58
threshold 0.01  precision 0.697  recall 1.000  F1 0.822  flagged 76

predict_proba returns one column per class; [:, 1] takes the probability of True (malignant). At a strict threshold of 0.9 every flag is correct (precision 1.000) but 10 of 53 cancers are missed. At a nervous threshold of 0.01 every cancer is caught (recall 1.000) but 23 healthy patients are sent for further tests. Nothing about the model changed — only the decision rule. Choosing that threshold is a business or clinical decision, not a statistical one.

Measuring regression models

When a model predicts a number — a house price, a delivery time, next month’s electricity use — there is no confusion matrix. Instead you measure how far the predictions land from the truth. Three measures cover most needs:

• Mean absolute error (MAE): the average size of the miss, ignoring direction. In the units of the target, so easy to explain: “we are off by RM 41 on average”.
• Root mean squared error (RMSE): square each miss, average, take the square root. Squaring punishes big misses much more than small ones, so RMSE is always at least as large as MAE, and a big gap between them means a few predictions are badly wrong.
• R² (coefficient of determination): the share of the target’s variation the model explains. 1.0 is perfect; 0.0 means no better than always predicting the mean; it can even go negative for a model worse than that.

from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error, root_mean_squared_error, r2_score

X, y = load_diabetes(return_X_y=True)   # y = disease progression one year later
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

model = LinearRegression().fit(X_train, y_train)
pred = model.predict(X_test)

print("Target range:", y.min(), "to", y.max(), " mean", round(y.mean(), 1))
print("MAE :", round(mean_absolute_error(y_test, pred), 1))
print("RMSE:", round(root_mean_squared_error(y_test, pred), 1))
print("R2  :", round(r2_score(y_test, pred), 3))

Target range: 25.0 to 346.0  mean 152.1
MAE : 41.5
RMSE: 53.4
R2  : 0.485

The diabetes dataset predicts a measure of disease progression from ten patient measurements. Printing the target’s range first gives the errors meaning: an average miss of 41.5 on a scale running from 25 to 346 is substantial, and R² of 0.485 says the model explains about half the variation. That is a typical, honest result for real medical data — and a reminder that “how good is good enough” depends on what the prediction will be used for. (root_mean_squared_error arrived in scikit-learn 1.4; older tutorials compute mean_squared_error(...) ** 0.5 instead.)

Overfitting, underfitting and cross-validation

A model that memorises its training data looks brilliant on that data and disappoints on anything new — overfitting. A model too simple to capture the pattern does badly everywhere — underfitting. The tell-tale sign is the gap between training and test scores. Decision trees make the effect easy to see, because max_depth controls how many questions the tree may ask:

from sklearn.datasets import load_breast_cancer
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)

for depth in [1, 2, 3, 5, 8, None]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=42).fit(X_train, y_train)
    print(f"max_depth={str(depth):<5} train {tree.score(X_train, y_train):.3f}"
          f"  test {tree.score(X_test, y_test):.3f}")

max_depth=1     train 0.923  test 0.923
max_depth=2     train 0.958  test 0.909
max_depth=3     train 0.977  test 0.944
max_depth=5     train 0.995  test 0.937
max_depth=8     train 1.000  test 0.923
max_depth=None  train 1.000  test 0.923

.score returns accuracy for classifiers. With one question (depth 1) the tree is too simple: train and test agree, but both are mediocre. As depth grows, training accuracy climbs to a perfect 1.000 — the tree has memorised every training example — while test accuracy peaks around depth 3 and then slips. A perfect training score is not a sign of success; it is a warning.

But notice how small these test differences are: 0.944 against 0.937 is a difference of one tumour in 143. With a single test set, you might be choosing depth 3 out of luck. Cross-validation fixes this by splitting the data into, say, five folds, training five times and testing each time on the fold that was held out. You get five scores, and their spread tells you how much to trust the average. Here we score by recall, because missed cancers are what we care about:

from sklearn.datasets import load_breast_cancer
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import StratifiedKFold, cross_val_score

X, y = load_breast_cancer(return_X_y=True)
y = (y == 0)                                  # positive = malignant
folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

for depth in [2, 3, 5, None]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=42)
    scores = cross_val_score(tree, X, y, cv=folds, scoring="recall")
    print(f"max_depth={str(depth):<5} recall per fold {scores.round(3)}"
          f"  mean {scores.mean():.3f} +/- {scores.std():.3f}")

max_depth=2     recall per fold [0.814 0.744 0.881 0.976 0.881]  mean 0.859 +/- 0.077
max_depth=3     recall per fold [0.93  0.814 0.857 0.976 0.857]  mean 0.887 +/- 0.058
max_depth=5     recall per fold [0.93  0.721 0.881 0.952 0.905]  mean 0.878 +/- 0.082
max_depth=None  recall per fold [0.907 0.744 0.81  0.952 0.881]  mean 0.859 +/- 0.074

StratifiedKFold keeps the share of malignant cases the same in every fold; shuffle=True with a fixed random_state makes the folds random but repeatable. scoring="recall" tells cross_val_score which metric to compute. Depth 3 still has the best average, but look at the spread: single folds range from 0.72 to 0.98. The honest summary is “depth 3 is probably a little better, and recall is roughly 0.89 give or take 0.06” — not “depth 3 has recall 0.944”. Finally, once you have chosen a model with cross-validation, keep a test set that played no part in the choosing, and look at it once at the end. Tuning against the test set until it looks good is just overfitting by another route.

Choosing a metric from the cost of mistakes

This guide’s framing Every metric is a hidden statement about which mistakes matter. So choose the metric before you train, by asking what each kind of error costs and who pays.

• Misses are expensive (cancer screening, fraud, safety defects): favour recall, and report precision so false alarms stay manageable.
• False alarms are expensive (spam filters that hide real mail, automated account suspensions, anything that accuses a person): favour precision.
• Both matter and the classes are imbalanced: F1, or better, look at the full precision–recall trade-off rather than one threshold.
• Numbers: MAE when every unit of error costs the same; RMSE when big misses are disproportionately bad; R² to compare models on the same data, never across different datasets.
• Groups: compute your metric separately for different groups (regions, languages, age bands). A model with good overall recall can still fail one group badly — a bias problem that the average hides.

When the costs can be put in money, you can go one step further and calculate it: if a missed fraud costs RM 2,000 and a false alarm costs RM 20 of staff time, count the FNs and FPs at each threshold and pick the cheapest. That turns a vague debate into arithmetic everyone can check.

Key terms on this page

Sources & further reading

• scikit-learn developers, Metrics and scoring: quantifying the quality of predictions (scikit-learn user guide, version 1.9).
• scikit-learn developers, Cross-validation: evaluating estimator performance (scikit-learn user guide).
• Google for Developers, Classification: accuracy, recall, precision and related metrics (Machine Learning Crash Course).
• Takaya Saito & Marc Rehmsmeier, “The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets”, PLOS ONE (2015).
• W. Nick Street, William H. Wolberg & Olvi L. Mangasarian, “Nuclear feature extraction for breast tumor diagnosis”, Proc. SPIE 1905 (1993) — the origin of the breast-cancer dataset.
• Gareth James, Daniela Witten, Trevor Hastie & Robert Tibshirani, An Introduction to Statistical Learning, 2nd edition (Springer, 2021; free online) — chapters on resampling and classification.