MLOps and Serving

A model in a notebook is an experiment; a model in production is a system that ages. This page covers the loop that keeps it healthy — versioning, tracking, deployment, monitoring, retraining — then the engineering of serving large models: batching, the KV cache, quantisation, and what a token really costs.

A credit-risk model is trained in 2019 and performs beautifully. In 2020 a pandemic changes who borrows, why, and who repays. Nobody touches the code; every test still passes; the dashboards are green. The model is simply answering questions about a world that no longer exists. Most failures of machine learning in production look like this — not a crash, but a quiet, gradual loss of truth. MLOps is the discipline of noticing, and of being able to fix it on a Tuesday afternoon rather than in a three-month rescue project.

The lifecycle: a loop, not a launch

MLOps borrows from DevOps the idea that building and running software are one job. What makes machine learning different is that its behaviour depends on data as much as on code, so data needs the same discipline as code: versions, reviews, tests and rollback. The Shipping an AI App page covered one release; this is the loop that follows. Step round it.

LOOPOne turn

None of these steps needs a particular product, and you can start with a spreadsheet and a naming convention. What matters is that each answers a question you will one day be asked in a hurry: what changed?

Drift: when the world moves

Two kinds of change break a deployed model. Data drift (or covariate shift) is when the inputs change: new customer segments, a new phone camera, a different hospital’s equipment. The relationship between input and answer may be the same, but the model is now asked about regions it saw little of in training. Concept drift is when the relationship itself changes: the same application details now predict default differently because the economy moved. Here are both, simulated with a random forest that learns a wavy decision boundary:

import numpy as np
from sklearn.ensemble import RandomForestClassifier
rng = np.random.default_rng(42)

def world(n, shift=0.0, new_rule=False):
    X = rng.normal(size=(n, 2))
    X[:, 0] += shift                                  # data drift: the inputs move
    y = (X[:, 1] > np.sin(2 * X[:, 0])).astype(int)   # the true rule: a wavy boundary
    if new_rule:
        y = (X[:, 1] > np.sin(2 * X[:, 0]) + 0.7).astype(int)  # concept drift
    return X, y

model = RandomForestClassifier(random_state=42).fit(*world(3000))
print(f"launch day          acc {model.score(*world(3000)):.3f}")
for s in [1.0, 2.0, 3.0]:
    print(f"data drift, +{s:.0f}     acc {model.score(*world(3000, shift=s)):.3f}")
print(f"concept drift       acc {model.score(*world(3000, new_rule=True)):.3f}")
launch day          acc 0.988
data drift, +1     acc 0.969
data drift, +2     acc 0.909
data drift, +3     acc 0.747
concept drift       acc 0.791

On launch day the model is 98.9% accurate. As the first feature drifts by one, two and three standard deviations, it is increasingly asked about territory it barely saw — a tree-based model simply extends its last known answer flat into the unknown — and accuracy falls to 74.7%. Concept drift is worse in one respect: the inputs look exactly as they did in training, so no input check can see it, yet accuracy drops to 79.1%. Only fresh labels reveal it.

Labels often arrive late — you learn whether a loan defaulted months later — so teams watch the inputs as an early warning. A common, simple statistic is the population stability index (PSI), which compares how a feature’s values fall into bins now versus at training time:

import numpy as np
rng = np.random.default_rng(42)

def psi(expected, actual, bins=10):
    """Population stability index between a reference and a live sample."""
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.histogram(expected, edges)[0] / len(expected)
    a = np.histogram(actual, edges)[0] / len(actual)
    e, a = np.clip(e, 1e-6, None), np.clip(a, 1e-6, None)
    return np.sum((a - e) * np.log(a / e))

reference = rng.normal(0, 1, 5000)                  # feature values at training time
for s in [0.0, 0.1, 0.5, 1.0]:
    live = rng.normal(s, 1, 5000)                   # this week's values
    print(f"shift {s:.1f}: PSI {psi(reference, live):.3f}")
shift 0.0: PSI 0.008
shift 0.1: PSI 0.015
shift 0.5: PSI 0.311
shift 1.0: PSI 1.028

A widely used rule of thumb in credit scoring treats PSI below 0.1 as stable, 0.1–0.25 as worth a look and above 0.25 as a significant shift — conventions, not laws. With no real shift the index is near zero (0.008, pure sampling noise); a half-standard-deviation shift crosses 0.25. Input monitoring tells you something moved; whether it matters needs labels or a careful look. For language models, the equivalent signals are changes in prompt topics and lengths, refusal rates, user ratings and the scores of a judge model on sampled traffic (see Evaluating Models).

Serving large models

Model serving for a large language model is unusual because generation happens one token at a time, and each token requires a pass through every layer. Two numbers describe a deployment, and they pull against each other:

The key fact behind most serving tricks: generating one token for one user is limited not by arithmetic but by memory bandwidth. To produce a token the GPU must read every weight from memory, and then it does relatively little maths with each. Reading the weights once and using them for many users’ tokens at the same time — batching — costs little extra time per step and multiplies throughput. Modern servers use continuous batching, adding new requests to the running batch as others finish rather than waiting to fill a batch. The trade-off is that bigger batches raise each user’s per-token latency a little.

The KV cache. In a Transformer, every new token attends to all earlier tokens through their key and value vectors (see Transformers in Depth). Recomputing those for the whole conversation at every step would be wasteful, so servers keep them in GPU memory — the KV cache — and compute keys and values only for the newest token. It saves recomputing the whole conversation at every step, but it costs memory that grows with every token of every concurrent conversation:

layers, width, bytes_per_number = 32, 4096, 2      # Llama-2-7B-like shape, 16-bit numbers
per_token = 2 * layers * width * bytes_per_number  # a key AND a value per layer
print(f"KV cache per token: {per_token / 2**20:.2f} MiB")
for ctx in [4_096, 32_768]:
    print(f"{ctx:>6,} tokens: {per_token * ctx / 2**30:.1f} GiB per conversation")
KV cache per token: 0.50 MiB
 4,096 tokens: 2.0 GiB per conversation
32,768 tokens: 16.0 GiB per conversation

Half a mebibyte per token, so a single long conversation can need more memory than the 7-billion-parameter model’s own 16-bit weights (about 14 GB). That is why long context is expensive, why newer architectures share keys and values across attention heads to shrink the cache, and why vLLM’s PagedAttention (Kwon et al., 2023) — managing the cache in fixed-size pages like an operating system’s virtual memory — raised throughput substantially by wasting less of it.

GPUs or CPUs? GPUs win for serving many users: thousands of parallel arithmetic units and memory bandwidth measured in terabytes per second. CPUs and laptops are perfectly usable for small or heavily quantised models at one user’s pace; tools such as llama.cpp and Ollama made running open-weight models locally routine As of 2026. For a team, the choice is usually between a hosted API, a managed open-model endpoint, or your own GPUs with a server such as vLLM — the arithmetic below helps decide.

Quantisation: fewer bits per weight

Quantisation stores weights (and sometimes activations) in fewer bits — 8 or 4 instead of 16 or 32. Since generation is limited by reading weights from memory, smaller weights mean faster tokens as well as a smaller footprint. The simplest scheme maps each float to an 8-bit integer with a single scale factor:

q = round( w ⁄ s ), s = max|w| ⁄ 127, ŵ = q · sw is an original weight, q its int8 code in −127 … 127, s the scale shared by the whole tensor, and ŵ the reconstructed weight used at inference.
import numpy as np
rng = np.random.default_rng(42)

W = rng.normal(0, 0.02, size=(512, 512)).astype(np.float32)   # one weight matrix
x = rng.normal(size=512).astype(np.float32)                    # one input vector

def quantise(w):                         # symmetric, one scale for the whole tensor
    scale = np.abs(w).max() / 127
    return np.round(w / scale).astype(np.int8), scale

q, scale = quantise(W)
W_back = q.astype(np.float32) * scale
print("bytes fp32:", W.nbytes, " int8:", q.nbytes)
print(f"scale {scale:.6f}   mean |weight error| {np.abs(W - W_back).mean():.2e}")
rel = np.linalg.norm(W @ x - W_back @ x) / np.linalg.norm(W @ x)
print(f"relative error of W @ x: {rel:.4%}")

W_out = W.copy(); W_out[0, 0] = 1.0      # a single outlier weight, 50x the typical size
q2, s2 = quantise(W_out)
rel2 = np.linalg.norm(W_out @ x - (q2.astype(np.float32) * s2) @ x) / np.linalg.norm(W_out @ x)
print(f"with one outlier: scale {s2:.6f}  relative error {rel2:.4%}  "
      f"weights rounded to zero {np.mean(q2 == 0):.0%}")
bytes fp32: 1048576  int8: 262144
scale 0.000789   mean |weight error| 1.97e-04
relative error of W @ x: 1.1359%
with one outlier: scale 0.007874  relative error 11.9876%  weights rounded to zero 16%

A quarter of the memory for about 1% error in the layer’s output — a good trade, and errors of this size usually move a model’s benchmark scores only slightly. But one outlier weight of 1.0 among values around 0.02 stretches the scale tenfold; the 255 available levels are now spread over a range the ordinary weights barely use, 16% of them round to zero, and the error rises to 12%. Real models do have such outliers, especially in activations, which is why practical methods use a separate scale per row or per small group of weights, or keep the rare outlier dimensions in higher precision (the idea behind LLM.int8(), Dettmers et al., 2022). For a 7-billion-parameter model, the weights alone take:

Always re-run your own evaluation set after quantising; averages can hide a task that degraded badly. The planned Expert page “Efficient Models” goes further into distillation and mixture-of-experts.

What a token costs

Whether to call an API or host a model yourself is mostly arithmetic, and the arithmetic is simple enough to do before any meeting. The figures below are example prices for illustration only — real prices vary widely by model and provider and change often As of 2026:

# EXAMPLE prices for the arithmetic only -- check your provider's pricing page
PRICE_IN, PRICE_OUT = 3.00, 15.00        # USD per million tokens (API)
requests_per_day = 10_000
tok_in, tok_out = 1_500, 300             # per request: prompt + retrieved text, answer

daily = requests_per_day * (tok_in * PRICE_IN + tok_out * PRICE_OUT) / 1e6
print(f"API: ${daily:,.2f} per day, ${daily * 30:,.0f} per 30 days")

GPU_PER_HOUR = 2.50                      # USD, one rented GPU (example)
tokens_per_second = 1_200                # output tokens/s with batching (example)
per_million = GPU_PER_HOUR / (tokens_per_second * 3600) * 1e6
print(f"self-hosted: ${per_million:.2f} per million output tokens at full load")
print(f"           ${per_million / 0.25:.2f} if the GPU is busy only 25% of the time")
API: $90.00 per day, $2,700 per 30 days
self-hosted: $0.58 per million output tokens at full load
           $2.31 if the GPU is busy only 25% of the time

Notice three things. First, input tokens dominate this API bill — 1,500 in for 300 out — so trimming retrieved context or using prompt caching often saves more than haggling over output prices. Second, self-hosting looks cheap per token only at high utilisation: a GPU you pay for by the hour costs the same whether busy or idle, and quiet nights and weekends quadruple the effective price in our example. Third, the comparison is not like for like — the self-hosted model must be good enough on your evaluation set, and someone must be paid to run it. For low or spiky traffic, APIs usually win; for steady, high volume with a model that meets the bar, owning the hardware can.

Reproducibility and hidden technical debt

In 2015 a team at Google led by D. Sculley published “Hidden Technical Debt in Machine Learning Systems”, still the best short read on why ML systems become hard to maintain. Its famous diagram shows the model code as a small box surrounded by much larger ones: data collection, feature extraction, configuration, serving infrastructure, monitoring. Consensus Its warnings have aged well:

“Changing Anything Changes Everything”

Inputs to a model are entangled: change the scaling of one feature, add a feature, or retrain an upstream model whose output you consume, and the behaviour of everything downstream can shift. There is no truly isolated change — hence evaluation on every change, not only on new models.

Glue code and pipeline jungles

Most of a real system is code that moves data between general-purpose packages and scripts that grew one special case at a time. The paper’s advice: keep the pipeline simple enough to rebuild from scratch, and treat it as first-class code with tests.

Hidden feedback loops

A model’s predictions change the world that produces its next training data: a recommender shows items, people click what they are shown, and the model learns that those items are popular. A loan model that rejects a group never sees how that group would have repaid.

Undeclared consumers and configuration debt

Other teams quietly start using your model’s outputs, so “small” changes break systems you did not know existed. Meanwhile, configuration — thresholds, feature lists, data paths — grows larger than the code and is rarely reviewed or tested.

Reproducibility is the everyday defence. A training run is reproducible when someone else can rebuild the same model from your records: pinned library versions (a lock file), fixed random seeds (as on every code page of this site), a data snapshot identified by hash, configuration in a versioned file rather than a notebook cell, and the evaluation results stored next to the model in the registry. GPU arithmetic is not always bit-for-bit deterministic, so aim for “the same within noise”, and know how big the noise is. For LLM applications, the same rule extends to prompts, retrieval indexes and the exact model version you call — record them with every release, because a provider’s model update is a change to your system even if you changed nothing.

Quick check

Inputs look exactly as they did at training time, but accuracy has fallen. What is the likeliest cause?

When the input distribution is unchanged but the input–label relationship has moved, only fresh labels reveal it.

Why does batching raise LLM throughput so much?

Token generation is memory-bandwidth bound; sharing one read of the weights across many users costs little extra time.

In the int8 demo, why did one outlier weight raise the error tenfold?

One scale for the whole tensor is set by its largest value; per-channel or per-group scales avoid this.

// Pause and reflect

  1. For a model you know (at work or in the news), what would data drift and concept drift look like in its particular world — and how long would the labels take to reveal them?
  2. If your provider silently updated the model behind your API tomorrow, how would you find out, and how quickly?
  3. Which of Sculley’s debts — entanglement, feedback loops, undeclared consumers, configuration — is most likely to bite your team first?

Your notes are saved privately in this browser.

Try this

Extend the drift simulation into a tiny monitoring loop: for 12 “weeks”, let the shift grow by 0.25 each week, compute PSI on the first feature and the model’s accuracy, and retrain on the latest week’s data whenever PSI exceeds 0.25. Print a weekly table. How much accuracy does the retraining rule save — and what would it cost if labels arrived four weeks late?

Key terms on this page

Sources & further reading