Fine-Tuning and Preference Training

A pretrained model can continue any text but answers nobody. This page follows the steps that turn it into an assistant — demonstrations, preferences, reward models, DPO and Constitutional AI — then shows how LoRA makes fine-tuning affordable, what it costs you in forgetting, and when you should not fine-tune at all.

Ask a freshly pretrained model “What is the capital of Malaysia?” and it may reply “What is the capital of Thailand? What is the capital of Indonesia?” — because on the web, a question is often followed by more questions from the same quiz. The model is not broken. It has learned exactly what pretraining asked of it: continue the text plausibly. Everything that makes an assistant feel like an assistant — answering the question, following instructions, declining some requests, admitting uncertainty — is added afterwards, by a comparatively tiny amount of further training. This page is about that last mile.

From base model to assistant

The process is usually called post-training. The reference recipe is the one OpenAI published for InstructGPT in 2022 (Ouyang et al.), building on earlier work on learning from human preferences by Christiano and colleagues (2017). It has three stages, and most later methods are variations on one or more of them. Tap each stage.

That last result is the key lesson of post-training. The capability was already in the base model; the tuning changed which of its possible behaviours it shows by default. A small, well-chosen dataset of behaviour moves a large model a long way.

Supervised fine-tuning on demonstrations

Supervised fine-tuning (SFT) is the simplest stage to understand because nothing new happens mathematically. You format each example as a conversation — a system message, a user turn, the ideal assistant turn — run it through the model, and compute the usual next-token cross-entropy loss. The only twist is a mask: the loss is counted on the assistant’s tokens but not on the prompt’s, because you want the model to learn to write answers, not to write questions. When the demonstrations are a broad mix of instructions, this is also called instruction tuning.

Three practical facts dominate SFT. Quality beats quantity: a few thousand carefully written examples often beat a scraped million, because the model imitates everything, including sloppiness. The learning rate is small — typically far smaller than in pretraining — since you are adjusting, not rebuilding. And a few epochs are enough; with small datasets the model quickly memorises and starts to overfit, which you will see as the validation loss turning upward exactly as on Measuring Models.

SFT has a ceiling. It can only teach behaviour someone can write down, and for many qualities — tone, judgement, which of two decent answers is better — it is far easier for a person to choose than to compose. That is where preferences come in.

Learning from preferences: reward models, RLHF and DPO

In reinforcement learning from human feedback, labellers see two (or more) answers to the same prompt and pick the better one. A reward model — usually a copy of the language model with its output layer replaced by a single number — is trained so that the chosen answer yw (“winner”) scores higher than the rejected one yl (“loser”). The standard loss comes from the Bradley–Terry model of paired comparisons (1952):

LRM = − log σ( r(x, yw) − r(x, yl) )where x is the prompt, r(x, y) the reward model’s score for answer y, and σ the sigmoid. σ of the score gap is the model’s probability that the human’s choice is the better answer; the loss is how surprised it is by the human’s actual choice.

Only the difference of scores matters, so rewards have no natural zero — a point worth remembering when someone quotes a reward value as if it meant something on its own. Here is the loss on four pairs:

import numpy as np

# reward-model scores for (chosen, rejected) answer pairs
r_chosen   = np.array([2.0, 0.5, 1.0, -0.3])
r_rejected = np.array([-1.0, 0.4, 2.0, -0.3])

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

p_agree = sigmoid(r_chosen - r_rejected)   # model's P(chosen beats rejected)
loss = -np.log(p_agree)                    # pairwise (Bradley-Terry) loss
for i in range(4):
    print(f"pair {i}: margin {r_chosen[i]-r_rejected[i]:+.1f}  "
          f"P(chosen wins) {p_agree[i]:.3f}  loss {loss[i]:.3f}")
print("mean loss:", round(loss.mean(), 3))
pair 0: margin +3.0  P(chosen wins) 0.953  loss 0.049
pair 1: margin +0.1  P(chosen wins) 0.525  loss 0.644
pair 2: margin -1.0  P(chosen wins) 0.269  loss 1.313
pair 3: margin +0.0  P(chosen wins) 0.500  loss 0.693
mean loss: 0.675

Pair 0 is already ranked correctly by a wide margin, so it contributes almost nothing. Pair 2 is ranked the wrong way round and dominates the loss; its gradient pushes the chosen score up and the rejected one down. A tie (pair 3) costs log 2 ≈ 0.693 — the loss of a coin flip.

With a reward model in hand, the language model (now called the policy) is optimised to produce answers that score highly, with a brake:

maximise E[ r(x, y) ] − β · KL( πθ ‖ πref )where πθ is the model being trained, πref the frozen SFT model, KL the Kullback–Leibler divergence (how differently the two models distribute probability over answers) and β how hard the brake is applied.

The brake matters because a reward model is only an approximation of human judgement, and an optimiser will find its blind spots — answers that score well but are not actually better, such as ever-longer or more flattering replies. Gao, Schulman and Hilton (2022) measured this “overoptimisation”: as the policy is pushed harder against a learned reward, the true quality first rises and then falls. It is Goodhart’s law in miniature, and one plausible ingredient of sycophancy.

RL with a separate reward model works, but it is fiddly: four models in memory (policy, reference, reward model and a value network for PPO), sampling inside the training loop, and many sensitive hyperparameters. Direct Preference Optimisation (Rafailov et al., 2023) showed that, for this particular objective, the optimal policy can be written in closed form — so you can skip the reward model and the RL loop and train directly on the preference pairs:

LDPO = − log σ( β · [ log πθ(yw|x) / πref(yw|x) − log πθ(yl|x) / πref(yl|x) ] )Each log ratio says how much more likely the model under training makes an answer than the frozen reference does. The loss rewards raising the preferred answer’s ratio relative to the rejected one’s; β plays the same braking role as before.

Compare it with the reward-model loss: the same −log σ of a difference, but the “reward” is now the model’s own log-probability ratio. That is the sense of the paper’s subtitle, “your language model is secretly a reward model”. In code it is a few lines — all you need are the summed token log-probabilities of each whole answer under both models:

import numpy as np

def dpo_loss(pol_w, pol_l, ref_w, ref_l, beta=0.1):
    """Log-probabilities of the preferred (w) and dispreferred (l) answers
    under the model being trained (pol) and the frozen reference (ref)."""
    margin = beta * ((pol_w - ref_w) - (pol_l - ref_l))
    return -np.log(1 / (1 + np.exp(-margin))), margin

# total log-probability of each whole answer (sums over its tokens)
ref_w, ref_l = -42.0, -40.0          # the reference slightly prefers the WORSE answer
for step, (pol_w, pol_l) in enumerate([(-42.0, -40.0), (-38.0, -43.0), (-30.0, -60.0)]):
    loss, m = dpo_loss(pol_w, pol_l, ref_w, ref_l)
    print(f"policy {step}: logp(w)={pol_w:6.1f} logp(l)={pol_l:6.1f}  margin {m:+.2f}  loss {loss:.3f}")
policy 0: logp(w)= -42.0 logp(l)= -40.0  margin +0.00  loss 0.693
policy 1: logp(w)= -38.0 logp(l)= -43.0  margin +0.70  loss 0.403
policy 2: logp(w)= -30.0 logp(l)= -60.0  margin +3.20  loss 0.040

At the start the policy is the reference, so the margin is zero and the loss is log 2, whatever the reference believed. Training lowers the loss by moving probability towards the preferred answer relative to where the reference had it. Notice policy 2 has pushed the rejected answer’s probability down enormously; in practice that is one known failure mode of DPO, which can lower the likelihood of both answers while widening the gap. Many variants since 2023 adjust for this.

Debated Whether DPO-style offline methods match well-tuned online RL is still argued. They are simpler and cheaper and widely used in open-model work; RL with fresh samples from the current policy can explore answers the preference data never contained, and has returned to prominence for training reasoning on tasks with checkable answers As of 2026.

Other views
  • “DPO is enough”: it optimises the same objective without the instability of RL, and many strong open models were tuned with it or its variants.
  • “Online RL still wins”: learning from samples the current model actually produces corrects its own mistakes; offline pairs only show it someone else’s.
  • “The data matters more than the algorithm”: several comparisons find that the quality and coverage of preference data move results more than the choice between methods.

Constitutional AI and AI feedback

Human preference labels are slow, expensive and — for harmful content — unpleasant work. Constitutional AI (Bai et al., Anthropic, 2022) replaces much of that labour with a written list of principles, the “constitution”, and the model itself. It has two phases:

The model answers a set of difficult prompts, is asked to critique its own answer against a randomly drawn principle (for example, to identify ways the answer is harmful), and then to rewrite it. The revised answers become SFT data. The model is, in effect, writing its own demonstrations, steered by text a person can read and argue with.

The broader idea — models generating or judging the training data for models — is now everywhere, from synthetic SFT data to the LLM judges you will meet on Evaluating Models. Exact current recipes at the large labs are mostly unpublished, so treat confident claims about them with care.

LoRA: fine-tuning on a budget

Full fine-tuning updates every parameter. For a 7-billion-parameter model trained with the Adam optimiser in 16-bit precision, the weights, gradients and optimiser state together need on the order of a hundred gigabytes of GPU memory, and each fine-tuned copy is another full-size file. Parameter-efficient fine-tuning trains a small number of new parameters instead and leaves the original weights frozen. The best known method is LoRA, Low-Rank Adaptation (Hu et al., 2021).

The bet behind LoRA is that the change a fine-tune needs to make to a weight matrix is simple, even if the matrix itself is not — that it can be well approximated by a matrix of low rank. So instead of learning a full update ΔW, LoRA learns two thin matrices whose product has that shape:

W′ = W0 + (α ⁄ r) · B Awhere W0 (d × k) is the frozen pretrained weight, B is d × r, A is r × k, the rank r is small (often 4–64), and α is a fixed scaling constant. Only A and B are trained. B starts at zero, so at step 0 the model is exactly the original.

A rank-r update costs r(d + k) numbers instead of d × k. Let us count, and then look at a small update we can actually print:

import numpy as np
rng = np.random.default_rng(42)

d, k, r = 4096, 4096, 8                    # one attention projection in a 7B-class model
full = d * k
lora = r * (d + k)                          # B is d x r, A is r x k
print(f"full update: {full:,} numbers   LoRA r={r}: {lora:,}  ({100*lora/full:.2f}%)")

# a tiny version we can print
d, k, r = 6, 6, 2
W = rng.normal(size=(d, k))                 # frozen pretrained weight
A = rng.normal(scale=0.5, size=(r, k))      # trainable, random start
B = np.zeros((d, r))                        # trainable, zero start
print("delta at start is all zero:", np.allclose(B @ A, 0))

B = rng.normal(scale=0.5, size=(d, r))      # pretend training has moved B
delta = B @ A                               # the learned update, 6 x 6
print(np.round(delta, 2))
print("rank of update:", np.linalg.matrix_rank(delta))
W_merged = W + (16 / r) * delta             # alpha / r scaling, merged for serving
print("merged shape:", W_merged.shape)
full update: 16,777,216 numbers   LoRA r=8: 65,536  (0.39%)
delta at start is all zero: True
[[-0.03 -0.14 -0.14  0.11  0.14  0.1 ]
 [-0.11 -0.02 -0.04  0.08  0.19  0.07]
 [ 0.09  0.29  0.29 -0.25 -0.34 -0.22]
 [ 0.12  0.06  0.08 -0.11 -0.23 -0.1 ]
 [-0.24  0.14  0.1   0.04  0.27  0.05]
 [-0.14  0.24  0.21 -0.09  0.05 -0.06]]
rank of update: 2
merged shape: (6, 6)

Hu et al. reported that on GPT-3 175B, LoRA cut trainable parameters by a factor of about 10,000 and GPU memory by about three compared with full fine-tuning, with comparable quality on their tasks. QLoRA (Dettmers et al., 2023) went further by keeping the frozen base in 4-bit precision (see quantisation on MLOps and Serving), making it possible to fine-tune a 65-billion-parameter model on a single 48 GB GPU. Libraries such as Hugging Face PEFT implement these methods in a few lines As of 2026.

Forgetting, and other risks

Gradient descent has no respect for what a network already knows. Train it only on new data and it will happily overwrite old skills: catastrophic forgetting, first described for neural networks by McCloskey and Cohen in 1989. You can watch it happen with a linear classifier on scikit-learn’s digits — learn digits 0–4, then train only on 5–9:

import numpy as np
from sklearn.datasets import load_digits
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import train_test_split

X, y = load_digits(return_X_y=True)
X = X / 16.0
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42)
old_tr, new_tr = ytr < 5, ytr >= 5          # task A: digits 0-4, task B: digits 5-9
old_te, new_te = yte < 5, yte >= 5

clf = SGDClassifier(loss="log_loss", random_state=42)
for epoch in range(20):                     # learn task A
    clf.partial_fit(Xtr[old_tr], ytr[old_tr], classes=np.arange(10))
print(f"after task A: old digits {clf.score(Xte[old_te], yte[old_te]):.3f}")

for epoch in range(20):                     # then train ONLY on task B
    clf.partial_fit(Xtr[new_tr], ytr[new_tr])
print(f"after task B: old digits {clf.score(Xte[old_te], yte[old_te]):.3f}  "
      f"new digits {clf.score(Xte[new_te], yte[new_te]):.3f}")
after task A: old digits 0.992
after task B: old digits 0.008  new digits 0.975

From 99% to under 1%: the model has not merely got worse at 0–4, it now labels almost all of them as 5–9, because nothing in the second phase told it those answers still existed. The standard remedy is replay — mix some old data into the new training. Replacing the second loop with one that trains on task B plus a random 10% of task A’s training examples:

# replay: task B plus a random 10% of task A (same script, second loop changed)
rng = np.random.default_rng(42)
keep = rng.choice(np.flatnonzero(old_tr), size=int(0.1 * old_tr.sum()), replace=False)
mix = np.concatenate([np.flatnonzero(new_tr), keep])
for epoch in range(20):
    clf.partial_fit(Xtr[mix], ytr[mix])
with replay: old digits 0.890  new digits 0.971

A large language model forgets less dramatically, but the same force acts on it. InstructGPT mixed pretraining data back into its RL stage for exactly this reason. Low learning rates, few epochs, LoRA’s restricted updates and a held-out test of general ability, not just your task, are the everyday defences. Other risks to plan for:

Prompt, retrieve or fine-tune?

This guide’s framing Fine-tuning is the most expensive lever to pull and the hardest to undo, so climb this ladder from the bottom and stop at the first rung that meets your evaluation set.

  1. Write a better promptClear instructions, a system prompt, a few worked examples. Minutes to change, free to try, and fixes more than people expect.
  2. Add retrievalIf the problem is missing or changing knowledge — your policies, today’s prices — use retrieval-augmented generation. Facts stay editable and citable.
  3. Use tools or a bigger modelCalculations, lookups and actions belong in code the model calls. A stronger model may simply solve it.
  4. Fine-tune with LoRAWhen you need a consistent format, tone or narrow skill across thousands of calls, want a smaller model to match a larger one on one task, or must cut prompt length and latency.
  5. Preference-tune or train fullyWhen behaviour itself must change and you have the data, compute and evaluation discipline to check you did not break something else.

A rule of thumb that survives most cases: RAG changes what the model knows at answer time; fine-tuning changes how it behaves. They combine well — a fine-tuned model that is better at using retrieved passages is a common design.

Quick check

Your support bot must quote this month’s fees, which change often. Best first move?

Changing facts belong in retrieval, where they can be updated and cited. Fine-tuning teaches behaviour, and teaches facts unreliably.

In the DPO loss, what is the reference model for?

The ratios against a frozen reference play the role the KL penalty plays in RLHF, stopping the policy from drifting arbitrarily far.

Why does LoRA initialise B to zero?

B A = 0 at step 0, so the adapted model begins identical to the original; A is random so gradients still reach B.

// Pause and reflect

  1. Think of a task you might fine-tune for. Which rung of the ladder did you skip, and what evidence would convince you to climb past it?
  2. A reward model is a learned proxy for human judgement. What behaviour would you expect an optimiser to find that scores well but that you would not actually want?
  3. Constitutional AI makes values explicit as text. Who should get to write such a document for a model used across Malaysia’s languages and communities?

Your notes are saved privately in this browser.

Try this

Write your own LoRA in NumPy for a linear regression: freeze a weight matrix fitted on load_diabetes, then learn a rank-1 update B A by gradient descent on a shifted target (for example, the target plus a linear function of two features). Compare the number of trained numbers and the test error against refitting the whole matrix — and try rank 2.

Key terms on this page

Sources & further reading