Pretraining and Scaling Laws

In 2022 a team at DeepMind trained a model with 70 billion parameters that beat their own 280-billion-parameter model, along with GPT-3 and several other larger rivals, on most of the tasks they measured. It used about the same amount of computation as the bigger model. The trick was not a new architecture or a clever loss. They had simply noticed that the field had been spending its compute badly: building models too large and feeding them too little text. That finding, now called the Chinchilla result, is a scaling law, and scaling laws are what turned pretraining from an expensive gamble into something closer to engineering. This page builds up to them from the objective that makes it all work.

The objective: predict the next token

You have met the intuition in What Is a Large Language Model?: the model reads text and guesses what comes next. Here is the precise version. Take a sequence of tokens x₁, …, x_T (produced by the tokeniser). A Transformer with a causal mask outputs, at every position t, a probability distribution over the vocabulary for the next token. Training minimises the average cross-entropy — the negative log-probability the model gave to the token that actually came next:

Three things make this objective remarkable. It is self-supervised: the labels are just the text shifted by one, so any text is training data and nobody needs to annotate anything. It is dense: thanks to the causal mask, a sequence of 2,048 tokens yields 2,048 predictions from one forward pass. And it is demanding: to predict the next word of a chemistry paper, a court judgment or a Python file well, a model benefits from representing chemistry, law and Python. Next-token prediction sounds like autocomplete, but minimising it on enough varied text rewards learning a great deal about the world that produced the text.

Two reference points help when reading loss curves. A model that guesses uniformly over a vocabulary of size V has loss ln V. And e^L, the perplexity, is the effective number of choices the model is hesitating between: a loss of 1.5 nats means it is, on average, as uncertain as if choosing among e^1.5 ≈ 4.5 equally likely tokens.

A tiny language model, trained

The smallest language model worth training is a bigram model: it predicts the next character from the current one alone. As a neural network it is a single V × V weight matrix whose row i holds the logits for “what follows character i”. That is exactly an embedding lookup followed by a softmax, trained with the same cross-entropy as a large model. Here it is on four sentences of Bahasa Melayu.

import numpy as np

text = ("pagi pagi ali pergi ke pasar membeli ikan dan sayur . "
        "petang petang siti pergi ke kedai membeli roti dan susu . "
        "malam malam kami makan nasi lemak dan minum teh tarik . "
        "esok pagi kami pergi ke sekolah dan belajar membaca . ")
chars = sorted(set(text))
stoi = {c: i for i, c in enumerate(chars)}
ids = np.array([stoi[c] for c in text])
x, y = ids[:-1], ids[1:]                      # each char predicts the next one
V = len(chars)

rng = np.random.default_rng(42)
W = rng.standard_normal((V, V)) * 0.01        # logits for next char, given this char

def sample(n=40, start="p"):
    out, i = start, stoi[start]
    for _ in range(n):
        p = np.exp(W[i] - W[i].max()); p /= p.sum()
        i = rng.choice(V, p=p); out += chars[i]
    return out

print(f"vocab {V}, training pairs {len(x)}, ln(V) = {np.log(V):.3f}")
print("before:", sample())
for step in range(301):
    logits = W[x]                             # (N, V): one row lookup per char
    p = np.exp(logits - logits.max(1, keepdims=True)); p /= p.sum(1, keepdims=True)
    loss = -np.log(p[np.arange(len(y)), y]).mean()   # cross-entropy
    grad = p.copy(); grad[np.arange(len(y)), y] -= 1  # d loss / d logits
    dW = np.zeros_like(W); np.add.at(dW, x, grad / len(y))
    W -= 10.0 * dW
    if step % 50 == 0:
        print(f"step {step:>3}  loss {loss:.3f}")
print("after: ", sample())
C = np.zeros((V, V)); np.add.at(C, (x, y), 1)  # plain counting, for comparison
P = C / C.sum(1, keepdims=True)
print(f"best possible bigram loss (from counts): {-np.log(P[x, y]).mean():.3f}")

vocab 22, training pairs 221, ln(V) = 3.091
before: pkkresja.s.egbep sshbmahyyreoohn ckbbsyls
step   0  loss 3.089
step  50  loss 1.679
step 100  loss 1.585
step 150  loss 1.555
step 200  loss 1.541
step 250  loss 1.533
step 300  loss 1.528
after:  pi lagi m dacakan . rgi paleli pemi r mba
best possible bigram loss (from counts): 1.506

Walk through it:

• Before training the weights are near zero, so every character is almost equally likely and the sample is random letters. The starting loss, 3.089, matches ln 22 = 3.091: uniform guessing, as the theory says.
• The gradient of softmax plus cross-entropy with respect to the logits is beautifully simple: the predicted probabilities minus 1 at the true token (grad[..., y] -= 1). np.add.at sums those gradients into the rows that were used — the same “only used rows learn” behaviour as an embedding table. The learning rate of 10 is large because the gradient is averaged over all pairs; see Backpropagation and Gradient Descent.
• The loss falls fast and then flattens near 1.53, close to 1.506, the best any bigram model can do on this text (computed by simply counting pairs). A bigram model can learn that p is often followed by e or a, but it cannot remember further back than one character.
• After training the sample has Malay-like syllables and spaces (“pi lagi m dacakan . rgi paleli”) but no words. To get further you need more context — an MLP over the last few characters, then attention over all of them. That progression, bigram → MLP → Transformer, is exactly the path of Karpathy’s “makemore” lectures, and every step lowers the loss on the same objective.

Scale this up by about twelve orders of magnitude in compute and you have pretraining. The code changes (PyTorch, GPUs, a Transformer, batches of millions of tokens) but the loop does not: forward pass, cross-entropy against the next token, backward pass, update. Practical details such as optimisers, learning-rate warm-up and decay, and mixed precision are covered in Training Well.

The data: crawls, curation and contamination

If the objective is simple, the data is where much of the craft lies. Most pretraining corpora start from web crawls — above all Common Crawl, a non-profit archive that has collected billions of web pages since 2008 — supplemented by code, books, scientific papers, encyclopaedias and, increasingly, synthetic text generated by other models As of 2026. A raw crawl is mostly unusable: menus, spam, boilerplate, machine-translated junk and endless duplicates. Turning it into training data is a pipeline:

One hazard deserves its own name. Data contamination happens when test material — the questions and answers of a benchmark — ends up in the training data, usually because someone posted it on the web. The model then scores well by recall rather than skill. The GPT-3 paper itself reported that a bug meant some overlaps with its benchmarks were not removed, and analysed the effect. Contamination is hard to rule out for any model trained on a large crawl, which is why careful evaluation (see Evaluating Models) prefers fresh, private or held-out test sets. Deduplication against benchmarks helps, but near-paraphrases slip through.

Openly documented datasets such as C4 (Raffel and colleagues, 2020), The Pile (Gao and colleagues, 2020) and FineWeb (Penedo and colleagues, 2024) let you inspect these choices. The datasets behind most frontier commercial models are not published, and their composition is rarely disclosed beyond broad description As of 2026. Questions of consent and copyright in training data are live legal disputes in several countries.

Counting compute

To reason about scale you need a unit. Training compute is counted in FLOPs — floating-point operations (multiplications and additions), the total, not per second. Consensus A standard approximation, used by Kaplan and colleagues in 2020 and nearly everyone since, is:

It ignores attention’s extra cost at long contexts and other details, so treat it as accurate to within tens of per cent for typical dense Transformers — good enough for budgeting, not for invoices.

def train_flops(n_params, n_tokens):
    return 6 * n_params * n_tokens            # ~2 forward + ~4 backward per param per token

runs = {"GPT-3 (2020)": (175e9, 300e9),
        "Chinchilla (2022)": (70e9, 1.4e12),
        "your 124M GPT-2 repro": (124e6, 10e9)}
for name, (N, D) in runs.items():
    print(f"{name:<22} N={N:.3g}  D={D:.3g}  C={train_flops(N, D):.2e} FLOPs  ({D / N:,.1f} tokens/param)")

# how long on hardware? assume 8 GPUs, each ~1e15 FLOP/s peak, 40% actually achieved
per_second = 8 * 1e15 * 0.40
for name, (N, D) in runs.items():
    hours = train_flops(N, D) / per_second / 3600
    print(f"{name:<22} {hours:>10,.1f} hours on 8 such GPUs")

GPT-3 (2020)           N=1.75e+11  D=3e+11  C=3.15e+23 FLOPs  (1.7 tokens/param)
Chinchilla (2022)      N=7e+10  D=1.4e+12  C=5.88e+23 FLOPs  (20.0 tokens/param)
your 124M GPT-2 repro  N=1.24e+08  D=1e+10  C=7.44e+18 FLOPs  (80.6 tokens/param)
GPT-3 (2020)             27,343.8 hours on 8 such GPUs
Chinchilla (2022)        51,041.7 hours on 8 such GPUs
your 124M GPT-2 repro         0.6 hours on 8 such GPUs

The 6·N·D estimate for GPT-3, 3.15 × 10²³, matches the 3.14 × 10²³ its paper reports. Chinchilla used almost twice as much compute but put it differently: a model 2.5 times smaller, trained on nearly five times as many tokens. The hardware assumption — about 10¹⁵ FLOP/s per GPU, roughly the 16-bit dense peak of a current data-centre accelerator As of 2026, at 40% utilisation — is illustrative. On eight such GPUs GPT-3 would take about three years (27,344 hours), which is why real runs use thousands of chips in parallel (covered in the Expert track). A GPT-2-small-sized reproduction on 10 billion tokens takes well under an hour on the same eight; that is the kind of run nanoGPT lets you do.

The largest runs are far bigger. Meta reported that its Llama 3 405B model used about 3.8 × 10²⁵ FLOPs on 15.6 trillion tokens (2024) — consistent with 6·N·D. Most frontier developers no longer disclose compute, parameters or token counts; independent estimates suggest the largest runs have passed 10²⁶ FLOPs, but these are informed guesses, not disclosures As of 2026. Regulators have adopted compute as a rough proxy for capability: the EU AI Act presumes a general-purpose model has “systemic risk” when its training compute exceeds 10²⁵ FLOPs As of 2026.

Scaling laws: Kaplan, then Chinchilla

In January 2020 Jared Kaplan and colleagues at OpenAI trained many language models across several orders of magnitude and found that test loss fell as a smooth power law in parameters, data and compute — straight lines on log–log plots — as long as none of the three was the bottleneck. Architecture details such as depth versus width mattered surprisingly little within a wide range. Their fits also implied how to spend a fixed compute budget: grow the model faster than the data (optimal model size rising roughly as C^0.73). GPT-3’s 175 billion parameters trained on only 300 billion tokens reflect that advice.

In 2022 Jordan Hoffmann and colleagues at DeepMind revisited the question with over 400 training runs, varying the learning-rate schedule to match each run’s length. Three different methods agreed that parameters and tokens should grow in roughly equal proportion with compute, which works out to roughly 20 training tokens per parameter. Their 70-billion-parameter Chinchilla, trained on 1.4 trillion tokens with the same compute as their 280-billion-parameter Gopher (300 billion tokens), outperformed it. Later analyses attributed much of the disagreement with Kaplan to experimental choices, notably learning-rate schedules and how parameters were counted. One of their methods fits a formula you can play with:

import numpy as np

# Hoffmann et al. (2022) fitted loss: L(N, D) = E + A / N**alpha + B / D**beta
E, A, B, alpha, beta = 1.69, 406.4, 410.7, 0.34, 0.28
loss = lambda N, D: E + A / N**alpha + B / D**beta

C = 5.88e23                                    # Chinchilla's budget, in FLOPs
for N in [10e9, 30e9, 70e9, 175e9, 500e9]:
    D = C / (6 * N)                            # tokens we can afford at this size
    print(f"N={N / 1e9:>4.0f}B  D={D / 1e9:>6,.0f}B tokens  "
          f"({D / N:>6.1f} per param)  predicted loss {loss(N, D):.3f}")

Ns = np.logspace(9, 12, 400)                   # search 1B .. 1T parameters
best = Ns[np.argmin(loss(Ns, C / (6 * Ns)))]
print(f"fit's optimum: {best / 1e9:.0f}B params, {C / (6 * best) / best:.0f} tokens per param")

N=  10B  D= 9,800B tokens  ( 980.0 per param)  predicted loss 1.946
N=  30B  D= 3,267B tokens  ( 108.9 per param)  predicted loss 1.930
N=  70B  D= 1,400B tokens  (  20.0 per param)  predicted loss 1.937
N= 175B  D=   560B tokens  (   3.2 per param)  predicted loss 1.962
N= 500B  D=   196B tokens  (   0.4 per param)  predicted loss 2.016
fit's optimum: 32B params, 93 tokens per param

This output is worth reading closely, because it teaches two lessons at once. First, the curve is flat near the optimum: at Chinchilla’s budget, anything from about 30 to 70 billion parameters predicts a loss within 0.01 of the best, while badly lopsided choices (175B on 560B tokens, GPT-3’s shape; or 500B parameters) clearly lose. Second, this fitted formula, with its published rounded constants, puts the optimum at about 32B parameters and 93 tokens per parameter — not 70B and 20. The famous “20 tokens per parameter” comes from the paper’s other two methods; the parametric fit was later re-examined by Besiroglu and colleagues at Epoch AI (2024), who reconstructed the data from the paper’s plots, found that the published fit conflicted with the other two methods, and traced the problem partly to how the fit was optimised and to rounding of the reported constants — exactly the rounded constants used above. Their corrected fit agrees with the other methods. Consensus The robust message is the direction — train smaller models on much more data than pre-2022 practice — and not any single constant.

Treat Chinchilla scaling as an empirical finding with caveats. It describes the compute-optimal trade-off for training loss, for one family of models and data. It ignores the cost of using the model afterwards. A smaller model that is cheaper to serve is often worth “overtraining” far past 20 tokens per parameter, and modern open models do exactly that:

• GPT-3 175B (2020)≈ 1.7 tokens / param
  300B tokens: Kaplan-era allocation.
• Chinchilla 70B (2022)20
  1.4T tokens: compute-optimal by design.
• Llama 3 405B (2024)≈ 39
  15.6T tokens.
• Llama 3 70B (2024)≈ 210
  About 15T tokens.
• Llama 3 8B (2024)≈ 1,900
  About 15T tokens: far past compute-optimal, because it is cheap to run.

Bar length is on a logarithmic scale (log of tokens per parameter); on a linear scale the first three bars would be almost invisible. Token counts are as reported by the developers.

Speculative How long the laws continue is an open question. Loss has kept falling predictably over many orders of magnitude, but high-quality human-written text is finite, and some researchers project that the largest training runs could exhaust the stock of public human text within the next several years; others expect synthetic data, multiple passes over data and new modalities to stretch the supply. Scaling laws also predict loss, not which skills a model will have — the subject of the next section.

Emergent abilities?

Debated In 2022 Jason Wei and colleagues catalogued emergent abilities: tasks on which small models score near chance and then, past some scale, performance jumps — multi-step arithmetic, some reasoning benchmarks, using a chain of thought. If abilities appear suddenly, smooth loss curves cannot forecast them, which matters for planning and for safety. In 2023 Rylan Schaeffer, Brando Miranda and Sanmi Koyejo argued that much of this is a mirage of measurement: an all-or-nothing metric such as exact match turns smooth underlying improvement into an apparent jump. You can reproduce their argument in a few lines:

import numpy as np

sizes = np.array([1e8, 3e8, 1e9, 3e9, 1e10, 3e10, 1e11])   # model sizes
# suppose per-token accuracy improves smoothly with log(size) ...
per_token = 0.60 + 0.13 * np.log10(sizes / 1e8)
# ... but the task is scored only if all 10 answer tokens are right
exact_match = per_token ** 10

for n, p, em in zip(sizes, per_token, exact_match):
    print(f"{n:8.0e} params  per-token {p:.2f}  exact-match {em:.3f}  {'#' * int(em * 40)}")

   1e+08 params  per-token 0.60  exact-match 0.006  
   3e+08 params  per-token 0.66  exact-match 0.016  
   1e+09 params  per-token 0.73  exact-match 0.043  #
   3e+09 params  per-token 0.79  exact-match 0.097  ###
   1e+10 params  per-token 0.86  exact-match 0.221  ########
   3e+10 params  per-token 0.92  exact-match 0.444  #################
   1e+11 params  per-token 0.99  exact-match 0.904  ####################################

Per-token accuracy rises steadily, by the same amount for each tenfold increase in size. But a ten-token answer only counts if every token is right, so the score is per-token accuracy to the tenth power: it sits near zero for most of the range and then shoots from 0.22 to 0.90 over the last tenfold step. Plotted on its own, that last column looks like an ability switching on. Nothing switched on; a smooth trend passed through a harsh metric.

What pretraining does not give you

A pretrained model — a base model — is a superb text continuer, and that is all it is trained to be. Give it “What is the capital of Malaysia?” and it may answer, or it may continue with three more quiz questions, because on the web questions often come in lists. It has no notion of being an assistant, no consistent persona, no reliable refusal of harmful requests and no particular commitment to the truth beyond what is likely to follow. GPT-3 showed that careful prompting with examples (“few-shot” prompting) can coax useful behaviour out of a base model, but the result is fragile.

The behaviour you know from chat assistants comes from post-training: supervised fine-tuning on examples of instructions and good responses, then preference training such as RLHF, as described in Fine-Tuning and Preference Training. Post-training uses a tiny fraction of the compute of pretraining but changes how the model behaves dramatically. A useful division of labour: pretraining supplies most of the knowledge and skill; post-training shapes which of it comes out, and how. Pretraining also fixes the model’s knowledge cutoff: nothing after the data was collected is in the weights.

Key terms on this page

Sources & further reading

• Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown et al., “Scaling Laws for Neural Language Models” (2020).
• Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch et al., “Training Compute-Optimal Large Language Models” (2022) — Chinchilla; NeurIPS 2022.
• Alec Radford, Karthik Narasimhan, Tim Salimans & Ilya Sutskever, “Improving Language Understanding by Generative Pre-Training” (2018); Radford et al., “Language Models are Unsupervised Multitask Learners” (2019).
• Tom B. Brown et al., “Language Models are Few-Shot Learners”, NeurIPS (2020) — GPT-3, including its contamination analysis.
• Jason Wei, Yi Tay, Rishi Bommasani et al., “Emergent Abilities of Large Language Models”, Transactions on Machine Learning Research (2022); Rylan Schaeffer, Brando Miranda & Sanmi Koyejo, “Are Emergent Abilities of Large Language Models a Mirage?”, NeurIPS (2023).
• Katherine Lee, Daphne Ippolito, Andrew Nystrom et al., “Deduplicating Training Data Makes Language Models Better”, ACL (2022).
• Tamay Besiroglu, Ege Erdil, Matthew Barnett & Josh You, “Chinchilla Scaling: A replication attempt” (2024); Llama Team, AI @ Meta, “The Llama 3 Herd of Models” (2024).
• Andrej Karpathy, nanoGPT and the “makemore” lectures in Neural Networks: Zero to Hero (2022 onwards).