Evaluating Models

Two models are tested on the same fifty questions. Model A scores 82%, model B 76%. The team ships A, writes a blog post about the six-point gain, and moves on. Run the numbers you will compute below and it turns out that two models with identical true ability, tested on fifty questions, differ by five points or more about half the time. The six points may be real. They may be a coin landing heads. Most of what goes wrong in model evaluation is not exotic; it is this — a confident number that cannot bear the weight put on it.

Why generative models are hard to measure

On Measuring Models you evaluated classifiers: each test example has one right label, and accuracy, precision and recall follow by counting. A language model breaks every part of that comfort.

• Many right answers. “Summarise this contract” has thousands of acceptable outputs and no single reference to compare against. Word-overlap scores such as BLEU and ROUGE, built for translation and summarisation, reward copying the reference’s wording more than being right.
• Randomness. The same prompt can give different answers. One run is one sample from a distribution, so a score needs repeated runs or at least an honest error bar.
• Sensitivity to format. Rewording the prompt, changing the answer letters or the number of examples shown can move a benchmark score by several points — sometimes more than the gap between models being compared.
• Behaviour, not just ability. We care whether a model is truthful, refuses the right things, admits uncertainty and is safe in the hands of a user who is trying to misuse it. None of these is a single number.
• The test may be in the training data. A model trained on much of the public internet may have seen the questions. More on that below.

That is why Liang and colleagues’ Holistic Evaluation of Language Models (HELM, 2022) argued for measuring many scenarios against seven kinds of metric — accuracy, calibration, robustness, fairness, bias, toxicity and efficiency — rather than crowning a model on one leaderboard column. You do not need HELM’s scale, but you do need its attitude: decide what you care about, then measure each part separately.

What the big benchmarks measure

Public benchmarks fall into three broad families. Each measures something real and something narrower than its name suggests.

Debated How far leaderboard positions reflect real usefulness is argued. In “The Leaderboard Illusion” (2025), Singh and colleagues reported that some providers tested many private model variants on Chatbot Arena and published only the best, inflating their rank; the arena’s organisers disputed parts of the analysis.

Contamination, saturation and pass@k

Contamination means test items, or close paraphrases, were in the training data. A model can then “answer” by recall rather than skill, and its score says little about new problems. With web-scale pretraining (see Pretraining and Scaling Laws) it is hard to rule out, because benchmarks are published, discussed and copied all over the web. Signs to watch for: suspiciously high scores on an old benchmark but not on a freshly written equivalent; a model completing a benchmark question word for word from its first half; or performance that drops on problems dated after the model’s knowledge cutoff. Defences include private held-out sets, regularly refreshed questions and canary strings that let builders filter benchmark files out of training data.

Saturation is the opposite end: when top models all score in the high 90s, remaining differences are mostly noise and label errors in the benchmark itself. A benchmark’s useful life is often only a few years.

For code, the standard metric needs a little care. If a model has k attempts at a problem, pass@k is the probability that at least one attempt passes the tests. The naive way to estimate it — run exactly k samples and check — is very noisy. Chen et al. (2021) instead generate n ≥ k samples, count the c correct ones, and compute the exact chance that a random k-subset contains at least one:

from math import comb

def pass_at_k(n, c, k):
    """Chance that at least one of k samples (drawn from n, c of them correct) passes."""
    if n - c < k:
        return 1.0
    return 1 - comb(n - c, k) / comb(n, k)

n, c = 20, 3                     # 20 attempts at one problem, 3 passed the tests
for k in [1, 5, 10]:
    print(f"pass@{k:<2d} = {pass_at_k(n, c, k):.4f}")
print("naive 1-(1-c/n)^5 =", round(1 - (1 - c / n) ** 5, 4))

pass@1  = 0.1500
pass@5  = 0.6009
pass@10 = 0.8947
naive 1-(1-c/n)^5 = 0.5563

pass@1 is simply 3/20. With five tries the chance of a pass jumps to 60%, and with ten to 89%. The last line shows the tempting shortcut of treating the attempts as independent draws with replacement: it gives 56%, not 60%, because sampling without replacement from a fixed set of 20 makes repeat failures less likely. Two lessons: always state k, and remember that a high pass@10 describes a workflow where something checks ten answers — not a model that is right the first time.

Models judging models

For open-ended answers, a cheap alternative to human raters is to ask a strong model to grade: LLM-as-judge. Zheng et al. (2023) found that a strong judge agreed with human experts about as often as humans agreed with each other — over 80% on their pairwise comparisons — which made the approach mainstream. They also documented its biases:

• Position bias: preferring whichever answer is shown first (or second), regardless of content.
• Verbosity bias: preferring longer answers, even when the extra length adds nothing.
• Self-preference: a tendency to favour answers written by the judge model itself or in its own style.
• Limited checking: a judge that cannot solve a maths problem cannot reliably grade the answer to it.

Mitigations are mostly good experimental hygiene: run each comparison twice with the order swapped and count only consistent verdicts; give the judge a concrete rubric and, where possible, a reference answer; ask for a short justification before the verdict; and, above all, check the judge against a sample of human labels before trusting it. Compare these two judge prompts:

The better prompt grades one answer against a reference instead of two answers against each other, which removes position bias; it names criteria and explicitly discounts length; and it asks for a parseable output your code can read. You still owe it a calibration check.

Building your own eval set

Consensus For any real application, a small evaluation set drawn from your own use case is worth more than any public leaderboard. The Builder track introduced the idea of an evaluation set on Shipping an AI App; here is how to make one that can bear decisions.

1. Start from real inputsCollect (anonymised, with permission) the questions users actually ask — including the short, misspelt and mixed Bahasa Melayu–English ones — not the tidy ones you imagine.
2. Write down what “good” meansA reference answer, a list of facts that must appear, a test that must pass, or a rubric. If two colleagues disagree on the grade, fix the rubric before you grade models.
3. Cover the awkward cases on purposeQuestions with no answer in your documents, off-topic and adversarial requests, personal data, edge cases from past incidents. Tag each item so you can report scores by category.
4. Freeze it, and keep a hidden partVersion the set. Keep a slice nobody looks at while tuning prompts, so you have one honest number at the end.
5. Report uncertaintyRun several samples per item if outputs vary, and always give a confidence interval — the next section shows how.
6. Grow it from failuresEvery bug report becomes a new test item. Over months the set comes to describe your product better than any document.

Statistical noise: why small evals mislead

An accuracy measured on n questions is an estimate, and its uncertainty shrinks only with the square root of n. To get a 95% confidence interval for a proportion, the textbook “±1.96 × √(p(1 − p)/n)” behaves badly near 0% and 100% and for small n. The Wilson score interval (Wilson, 1927) is a better default; the bootstrap — resample your results with replacement thousands of times and see how much the mean moves — is a general alternative that works for any metric.

import numpy as np

def wilson(correct, n, z=1.96):              # 95% Wilson score interval
    p = correct / n
    centre = (p + z**2 / (2 * n)) / (1 + z**2 / n)
    half = z * np.sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / (1 + z**2 / n)
    return centre - half, centre + half

def bootstrap(results, reps=10_000, seed=42):
    rng = np.random.default_rng(seed)
    means = rng.choice(results, size=(reps, len(results))).mean(axis=1)
    return np.percentile(means, [2.5, 97.5])

for correct, n in [(41, 50), (820, 1000)]:
    lo, hi = wilson(correct, n)
    results = np.array([1] * correct + [0] * (n - correct))
    blo, bhi = bootstrap(results)
    print(f"{correct}/{n} = {correct/n:.0%}   Wilson {lo:.3f}-{hi:.3f}   bootstrap {blo:.3f}-{bhi:.3f}")

41/50 = 82%   Wilson 0.692-0.902   bootstrap 0.700-0.920
820/1000 = 82%   Wilson 0.795-0.843   bootstrap 0.796-0.843

Both lines report 82%. On 50 questions the plausible range is roughly 69% to 90% — twenty points wide. On 1,000 it narrows to under five. The two methods agree closely, which is reassuring; the bootstrap is the one to reach for when your metric is not a simple proportion. Now the scenario from the opening, simulated directly: two models with the same true accuracy of 80%, each tested on n questions, ten thousand times.

import numpy as np
rng = np.random.default_rng(42)

true_acc, trials = 0.80, 10_000          # two models with the SAME true accuracy
for n in [50, 200, 1000]:
    a = rng.binomial(n, true_acc, size=trials) / n
    b = rng.binomial(n, true_acc, size=trials) / n
    gap = np.abs(a - b)
    print(f"{n:5d} questions: gap of 5+ points in {np.mean(gap >= 0.05):.0%} of runs, "
          f"largest gap {gap.max():.0%}")

   50 questions: gap of 5+ points in 53% of runs, largest gap 30%
  200 questions: gap of 5+ points in 21% of runs, largest gap 17%
 1000 questions: gap of 5+ points in 0% of runs, largest gap 7%

With 50 questions, equal models differ by five points or more in more than half of all runs, and occasionally by thirty. Even 200 questions leave a one-in-five chance of a five-point phantom gap. When both models answer the same questions, a paired comparison — counting only the items where they disagree — is more sensitive than comparing two overall scores, and it is what you should use for A/B decisions. The general rule: before you celebrate a difference, ask how big a difference noise alone would produce.

Goodhart’s law and the limits of scores

The economist Charles Goodhart made the original observation about monetary targets in 1975; Strathern’s compact version is the one everyone quotes. Goodhart’s law haunts machine learning because optimisation is what we do. Tune prompts against a fixed eval set for weeks and you will overfit it, exactly as a model overfits training data. Train against a learned reward and the policy finds the reward model’s blind spots. Compete on a public leaderboard and the field collectively overfits the benchmark — through contamination, selective reporting, or simply thousands of design choices each kept because it nudged the number up.

This guide’s framing No score is the truth about a model; each is a narrow, noisy view. Use several kinds of evidence that fail differently — automatic metrics, a calibrated judge, human review of samples, and live monitoring after release (see MLOps and Serving) — and trust conclusions that all of them support.

Key terms on this page

Sources & further reading

• Percy Liang, Rishi Bommasani, Tony Lee et al., “Holistic Evaluation of Language Models” (2022; Transactions on Machine Learning Research, 2023).
• Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song & Jacob Steinhardt, “Measuring Massive Multitask Language Understanding” (2020; ICLR 2021).
• Mark Chen, Jerry Tworek, Heewoo Jun et al., “Evaluating Large Language Models Trained on Code” (2021) — HumanEval and the unbiased pass@k estimator.
• Lianmin Zheng, Wei-Lin Chiang, Ying Sheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” (NeurIPS Datasets and Benchmarks, 2023); Wei-Lin Chiang et al., “Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference” (2024).
• Shivalika Singh et al., “The Leaderboard Illusion” (2025).
• Carlos E. Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” (2023; ICLR 2024).
• Edwin B. Wilson, “Probable inference, the law of succession, and statistical inference”, Journal of the American Statistical Association 22 (1927); Marilyn Strathern, “‘Improving ratings’: audit in the British University system”, European Review 5 (1997).