The Maths You Need

Open any deep-learning paper and the first page of equations can feel like a locked door: bold letters, Greek symbols, little ∂s and ∇s. Here is the secret most teachers leave until too late. Almost every one of those equations is built from about six ideas, and you already use most of them in NumPy without calling them by name. When you write X @ W, you are doing linear algebra. When you check whether a loss went down after a small change, you are doing calculus. When you turn scores into percentages that add up to 100, you are doing probability. This page puts names, formulas and intuition on things your code already does.

How much maths is “enough”?

This guide’s framing To use and debug neural networks you need to read the maths fluently, not prove theorems. Concretely, by the end of this page you should be able to do four things: predict the shape of any matrix product, explain what a gradient tells you, apply the chain rule to a function inside a function, and say why a classifier is trained with cross-entropy. That is enough for the next two pages, Neural Networks from Scratch and Backpropagation and Gradient Descent, and for most of this track.

What you can safely postpone: proofs, integration techniques, differential equations, measure theory, and most of a statistics degree. Eigenvalues and the singular value decomposition will matter later, for compression and interpretability, but not today. Research work needs more — optimisation theory, information theory, serious probability — and the books in the sources list will take you there when you need them.

One habit matters more than any single topic: check maths with code. Every formula below is followed by a small program that tests it numerically. If your derivation and your program disagree, one of them is wrong, and finding out which is the fastest way to learn. Examples were run with Python 3.13 and NumPy 2.5 As of 2026.

Before we begin, a decoder for the symbols you will meet. Tap each one.

• 1∑Sum“Add up all of these.” ∑_i w_ix_i means w₁x₁ + w₂x₂ + … — in NumPy, (w * x).sum().
• 2ᵀTransposeFlip a matrix so rows become columns. A (3, 2) matrix becomes (2, 3). In NumPy, W.T.
• 3∂Partial derivative∂L/∂w: how fast L changes when you nudge only w and hold everything else still.
• 4∇Gradient∇L (“nabla L” or “grad L”): the vector of all partial derivatives of L, one per parameter.
• 5σSigmoidσ(z) = 1 / (1 + e^−z): squashes any number into (0, 1). In statistics, σ can also mean a standard deviation — context tells you which.
• 6ŷPrediction“y-hat”: the model’s guess, as opposed to y, the true answer.

Vectors and the dot product

A vector is an ordered list of numbers. That is all it is in code: a one-dimensional NumPy array. The interesting part is the two ways of picturing it. You can see a vector as a point in space — the vector (3, 1) sits three steps right and one step up — or as an arrow from the origin to that point, with a direction and a length. Machine learning uses both pictures constantly. A row of features is a point; an embedding is an arrow whose direction carries meaning.

The single most important operation on two vectors is the dot product: multiply them element by element and add up the results.

The two halves of that formula are the two pictures. The left half is arithmetic you can do by hand. The right half says the result is large when the arrows are long and point the same way, zero when they are at right angles, and negative when they point apart. Divide out the lengths and you are left with cos θ — the cosine similarity you met in Embeddings and Semantic Search. Here it is on some made-up “taste vectors”, each dish scored 0–5 for spicy, sweet, rich (santan) and rice-based:

import numpy as np

# each dish scored 0-5 for: spicy, sweet, rich (santan), rice-based
nasi_lemak = np.array([4, 1, 4, 5])
nasi_kerabu = np.array([2, 1, 3, 5])
cendol     = np.array([0, 5, 4, 0])

def cosine(a, b):
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

print("dot(nasi lemak, nasi kerabu):", nasi_lemak @ nasi_kerabu)
print("dot(nasi lemak, cendol):     ", nasi_lemak @ cendol)
print("length of nasi lemak:", round(np.linalg.norm(nasi_lemak), 3))
print("cosine(nasi lemak, nasi kerabu):", round(cosine(nasi_lemak, nasi_kerabu), 3))
print("cosine(nasi lemak, cendol):     ", round(cosine(nasi_lemak, cendol), 3))

dot(nasi lemak, nasi kerabu): 46
dot(nasi lemak, cendol):      21
length of nasi lemak: 7.616
cosine(nasi lemak, nasi kerabu): 0.967
cosine(nasi lemak, cendol):      0.431

In NumPy, @ between two 1-D arrays is the dot product: 4×2 + 1×1 + 4×3 + 5×5 = 46. np.linalg.norm is the length, √(16 + 1 + 16 + 25) ≈ 7.616. The two rice dishes point in almost the same direction (cosine 0.967); the dessert shares only the santan (0.431). The scores are invented, but the mechanism is exactly how semantic search ranks documents.

Now the connection that matters for the next page. A neuron computes w · x + b. So a neuron is a similarity detector: its output is large when the input points in the same direction as its weight vector. Learning a neuron’s weights means learning which direction in input space it should respond to.

Matrices: transformations and batches

A matrix is a rectangular grid of numbers — a 2-D array. Again there are two pictures, and you need both.

Picture one: a matrix is a function. Multiplying a vector by a matrix moves it somewhere else: it can rotate, stretch, squash, shear or project it. Each output entry is a dot product of one row of the matrix with the input vector. The rotation below turns the point (3, 1) a quarter-turn anticlockwise to (−1, 3).

Picture two: a matrix is a batch. Stack a hundred feature vectors as the rows of a matrix X, and a single product X @ W pushes all hundred through the same transformation at once. This is why GPUs matter for deep learning: one large matrix product is many independent dot products that can run in parallel.

That last condition is the shape rule, and it is worth more than any theorem on this page: (n, k) @ (k, m) → (n, m). The inner sizes must match and disappear; the outer sizes survive. When you read (batch, features) @ (features, outputs), you should see (batch, outputs) without thinking.

import numpy as np

# a matrix is a function: this one rotates 2-D points by 90 degrees anticlockwise
R = np.array([[0, -1],
              [1,  0]])
p = np.array([3, 1])
print("R @ p =", R @ p)

# batched computation: 4 examples, 3 features each -> 2 outputs each
rng = np.random.default_rng(42)
X = rng.normal(size=(4, 3))      # (batch, features)
W = rng.normal(size=(3, 2))      # (features, outputs)
b = np.zeros(2)                  # one bias per output
Y = X @ W + b
print("X", X.shape, "@ W", W.shape, "->", Y.shape)

try:
    W @ X                         # wrong order: (3,2) @ (4,3)
except ValueError as e:
    print("Shape error:", e)

R @ p = [-1  3]
X (4, 3) @ W (3, 2) -> (4, 2)
Shape error: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 4 is different from 2)

Three details deserve a second look. First, X @ W + b adds a vector of length 2 to a (4, 2) matrix; NumPy’s broadcasting copies the bias onto every row, which is exactly what we want — every example gets the same bias. Second, the error message is ugly but readable once you know the shape rule: the inner sizes were 2 and 4, and they must match. Third, the order matters: AB is usually not the same as BA, and often only one of them is even defined. When a network crashes, a shape mismatch is the most likely culprit, so print shapes freely.

Two shape facts you will use daily. A transpose flips a matrix: if W is (3, 2), Wᵀ is (2, 3). And a whole layer of a neural network is the line X @ W + b followed by a non-linear function — so once you can read shapes, you can read networks.

Derivatives, gradients and the chain rule

Training a model means asking, over and over: “if I change this number a little, does the error go up or down, and how fast?” That question has a name. A derivative is the slope of a function at a point — the rate at which the output changes per unit change of the input, for very small changes.

Calculus gives exact rules for slopes: the derivative of x³ is 3x², the derivative of a constant is 0, and so on. But you can always estimate a slope by nudging the input and watching the output. That gives you a referee:

def f(x):
    return x**3 - 2*x

def f_prime(x):                  # derivative worked out by hand: 3x^2 - 2
    return 3*x**2 - 2

x = 1.5
print("analytic slope:", f_prime(x))
for h in [0.1, 0.01, 0.001]:
    forward = (f(x + h) - f(x)) / h
    central = (f(x + h) - f(x - h)) / (2*h)
    print(f"h={h:<6} forward {forward:.6f}   central {central:.6f}")

analytic slope: 4.75
h=0.1    forward 5.210000   central 4.760000
h=0.01   forward 4.795100   central 4.750100
h=0.001  forward 4.754501   central 4.750001

The hand-derived slope at x = 1.5 is 3(2.25) − 2 = 4.75. Both numerical estimates converge on it as h shrinks, but the central difference — nudging both ways — is far more accurate for the same h. On the backpropagation page we use exactly this trick to check every gradient of a neural network. (Do not make h absurdly small: below about 10⁻⁸, floating-point rounding starts to swamp the signal.)

Many inputs: partial derivatives and the gradient. A network’s loss depends on thousands or billions of weights. A partial derivative ∂f/∂w₁ is the slope in one direction only: nudge w₁, hold every other input fixed. Collect one partial derivative per input into a vector and you have the gradient.

If the gradient points uphill, then stepping a little in the opposite direction goes downhill. That one sentence is gradient descent.

import numpy as np

def f(w):                        # a bowl with its lowest point at (3, -1)
    return (w[0] - 3)**2 + 2*(w[1] + 1)**2

def grad_f(w):                   # one partial derivative per input
    return np.array([2*(w[0] - 3), 4*(w[1] + 1)])

def numeric_grad(f, w, h=1e-5):
    g = np.zeros_like(w)
    for i in range(len(w)):
        step = np.zeros_like(w); step[i] = h
        g[i] = (f(w + step) - f(w - step)) / (2*h)
    return g

w = np.array([0.0, 0.0])
print("f(w) =", f(w))
print("analytic gradient:", grad_f(w))
print("numeric gradient: ", numeric_grad(f, w).round(6))
w_new = w - 0.1 * grad_f(w)      # one small step against the gradient
print("after one step: w =", w_new, " f(w) =", round(f(w_new), 4))

f(w) = 11.0
analytic gradient: [-6.  4.]
numeric gradient:  [-6.  4.]
after one step: w = [ 0.6 -0.4]  f(w) = 6.48

At (0, 0) the gradient is (−6, 4): the bowl rises fastest if you decrease w₁ and increase w₂. So we do the opposite — add 0.6 to w₁, subtract 0.4 from w₂ — and the function drops from 11 to 6.48. The 0.1 is a learning rate. The numeric_grad function nudges each input in turn; it works for any function, but needs two evaluations per input, which is hopeless for a billion weights. That is why we need a cleverer method.

Functions inside functions: the chain rule. A neural network is a function of a function of a function: a matrix product, then an activation, then another matrix product, then a loss. The chain rule says how slopes compose.

Intuition: if g doubles any small change in x, and f triples any small change in g, then x moves y six times as much. Rates multiply along a chain. Here is the rule on a tiny neuron, h(x) = σ(3x + 1), using the fact that the sigmoid’s slope is σ(z)(1 − σ(z)):

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def h(x):                        # h(x) = sigmoid(3x + 1): a function inside a function
    return sigmoid(3*x + 1)

def h_prime(x):                  # chain rule: outer slope (at the inner value) x inner slope
    s = sigmoid(3*x + 1)
    return s * (1 - s) * 3

x = 0.2
numeric = (h(x + 1e-5) - h(x - 1e-5)) / 2e-5
print("chain rule:", round(h_prime(x), 8))
print("numeric:   ", round(numeric, 8))

chain rule: 0.41929138
numeric:    0.41929138

Eight decimal places of agreement. Backpropagation is nothing more than this rule applied to a long chain, organised so that every intermediate slope is computed once and reused. Keep this example in mind: the same three factors — outer slope, activation slope, inner slope — appear in every layer of every network.

Probability: softmax and cross-entropy

A classifier should not just say “cat”; it should say how sure it is. A probability distribution over a set of outcomes is a list of non-negative numbers, one per outcome, that add up to 1. A fair die gives each face 1/6. A language model gives each of its tens of thousands of possible next tokens a probability.

The expectation (or expected value) of a quantity is its probability-weighted average: what you would get on average over many draws. For a fair die, ∑ p(x) × x = 3.5 — a value the die can never actually show, which is a useful reminder that an average is not a typical outcome. Notice that an expectation is just a dot product of a probability vector with a value vector. Linear algebra keeps turning up.

A network’s last layer produces raw scores, called logits, which can be any real numbers. The softmax function turns them into a distribution:

How do we score a predicted distribution against the truth? The standard answer is the cross-entropy loss: take the probability the model gave to the correct answer, and compute its negative logarithm.

Why this formula? Because of maximum likelihood: a good model is one under which the data we actually observed is as probable as possible. The probability of a whole dataset of independent examples is the product of the probabilities of each; taking the log turns that product into a sum, and the minus sign turns “maximise” into “minimise”. So minimising average cross-entropy is the same as maximising the log-likelihood of the training data. When people say a language model is trained to predict the next token, this is the precise meaning.

import numpy as np

# expectation: the probability-weighted average of the outcomes
faces = np.arange(1, 7)
probs = np.full(6, 1/6)
print("expected value of a fair die:", probs @ faces)

def softmax(z):
    z = z - z.max()              # subtracting a constant changes nothing but avoids overflow
    e = np.exp(z)
    return e / e.sum()

logits = np.array([2.0, 1.0, 0.1])          # raw scores for: cat, dog, bird
p = softmax(logits)
print("softmax:", p.round(3), " sum =", p.sum())

true_class = 0                               # the picture really is a cat
print("cross-entropy if right class is cat: ", round(-np.log(p[true_class]), 3))
print("cross-entropy if right class is bird:", round(-np.log(p[2]), 3))
print("naive exp of a big logit:", np.exp(np.array([1000.0])))
print("stable softmax of big logits:", softmax(np.array([1000.0, 999.0])).round(3))

expected value of a fair die: 3.5
softmax: [0.659 0.242 0.099]  sum = 1.0
cross-entropy if right class is cat:  0.417
cross-entropy if right class is bird: 2.317
naive exp of a big logit: [inf]
stable softmax of big logits: [0.731 0.269]

Scores of 2.0, 1.0 and 0.1 become 66%, 24% and 10%. If the picture really is a cat, the loss is a modest 0.417; if it was a bird — to which the model gave only 10% — the loss is 2.317, more than five times larger. Cross-entropy punishes confident mistakes hard, which is exactly the pressure you want during training.

The last two lines show a practical trap. e¹⁰⁰⁰ is too big for a 64-bit float, so NumPy returns inf (and prints RuntimeWarning: overflow encountered in exp). Subtracting the largest score first changes nothing mathematically — the shift cancels between top and bottom of the fraction — but keeps every exponent at or below zero. Every serious library does this inside its softmax. If you have read about temperature in chat models, it is a number the logits are divided by before the softmax: below 1 sharpens the distribution, above 1 flattens it.

Logs and exponentials

Two functions do a lot of quiet work in machine learning, and they undo each other: the exponential e^x and the natural logarithm log x (NumPy’s np.log, base e ≈ 2.718). Four facts cover nearly every use:

• e^x is always positive and grows explosively. That is why softmax uses it: any score becomes a positive weight, and differences between scores become ratios.
• log turns products into sums: log(ab) = log a + log b. Sums are easier to differentiate and far kinder to computers.
• log of a probability is never positive: log 1 = 0, and log p heads to −∞ as p approaches 0. Hence the minus sign in cross-entropy.
• log is increasing, so the weights that maximise a likelihood also maximise its log. Taking logs never changes where the best answer is.

import numpy as np

rng = np.random.default_rng(42)
p = rng.uniform(0.05, 0.5, size=1000)   # 1,000 independent probabilities

print("product of probabilities:", np.prod(p))
print("sum of log-probabilities:", round(np.log(p).sum(), 2))
print("log turns products into sums:", round(np.log(0.2 * 0.5), 4), "=", round(np.log(0.2) + np.log(0.5), 4))

product of probabilities: 0.0
sum of log-probabilities: -1445.93
log turns products into sums: -2.3026 = -2.3026

Multiply a thousand modest probabilities and the answer is about e⁻¹⁴⁴⁶, far smaller than the tiniest number a 64-bit float can hold, so it underflows to exactly 0.0 and all information is lost. The sum of logs, −1445.93, is a perfectly ordinary number. That is the practical reason models work with log-probabilities everywhere, and why a sentence’s probability under a language model is reported as a sum of per-token log-probabilities.

A study plan

If parts of this page went too fast, that is normal. Here is a plan that has worked for many self-taught engineers; it assumes three to five hours a week and keeps code beside every idea. Tap through the steps.

1. Weeks 1–2 · Vectors and matricesWatch 3Blue1Brown’s Essence of Linear Algebra chapters on vectors, linear transformations, matrix multiplication and the dot product. After each, reproduce one picture in NumPy. Drill shapes until (n, k) @ (k, m) is automatic.
2. Weeks 3–4 · Derivatives and the chain rule3Blue1Brown’s Essence of Calculus on derivatives and the chain rule; then chapter 5 (Vector Calculus) of Mathematics for Machine Learning. Write a numeric_grad function and use it to check every derivative you work out by hand.
3. Weeks 5–6 · ProbabilityChapter 6 of Mathematics for Machine Learning, or chapter 3 of Goodfellow, Bengio and Courville’s Deep Learning. Implement softmax and cross-entropy yourself; reproduce the overflow and the underflow from this page and fix them.
4. Weeks 7–8 · Put it togetherRead the next two pages, then follow Andrej Karpathy’s micrograd video and type the code yourself. You will meet every idea on this page again — in a form you can run.
5. Afterwards · Fill gaps on demandWhen a paper uses something new (eigenvectors, KL divergence, Jacobians), learn it then, with a concrete use in front of you. Maths sticks best when it answers a question you already have.

Key terms on this page

Sources & further reading

• Marc Peter Deisenroth, A. Aldo Faisal & Cheng Soon Ong, Mathematics for Machine Learning (Cambridge University Press, 2020) — free PDF; chapters 2 (linear algebra), 5 (vector calculus) and 6 (probability).
• Ian Goodfellow, Yoshua Bengio & Aaron Courville, Deep Learning (MIT Press, 2016) — part I, chapters 2–4: linear algebra, probability and numerical computation (including the softmax overflow trick).
• Grant Sanderson (3Blue1Brown), Essence of Linear Algebra (video series, 2016) and Essence of Calculus (video series, 2017).
• Andrej Karpathy, Neural Networks: Zero to Hero (video course, 2022 onwards) — begins with derivatives and the chain rule in plain Python.
• Charles R. Harris et al., “Array programming with NumPy”, Nature 585 (2020) — and the NumPy broadcasting guide.