The Advanced Road

Strip away the chat window, the safety filters and the data centre, and a large language model is a long list of numbers and a short recipe for multiplying them. The recipe fits on a few pages. The numbers — billions of them — were found by a procedure you can write yourself in an afternoon. That is the strange promise of this track: the most talked-about technology of the decade is built from ideas a determined learner can understand completely. Not quickly, but completely.

Who this track is for

The Advanced track is for builders and engineers who can already write Python and have used libraries such as NumPy and scikit-learn — roughly, people who have finished the Builder track. Here you stop treating models as black boxes. You will implement a neural network and backpropagation from scratch, read the internals of the Transformer, and learn how models are pretrained, fine-tuned, evaluated and served.

If you want to understand why a model hallucinates, why longer context costs more, or why a fine-tune made things worse, this is where the answers live.

The map: what’s inside a modern model

Think of a modern model as a set of nested layers. Peel them one at a time, from the product you use down to the single idea at the centre. Each layer corresponds to pages in this track.

The thirteen pages after this one follow the layers roughly from the core outwards: The Maths You Need, Neural Networks from Scratch, Backpropagation and Gradient Descent, Training Well, Convolutional Networks, RNNs, LSTMs and Sequence Models, Transformers in Depth, Tokenisation and Embeddings, Pretraining and Scaling Laws, Fine-Tuning and Preference Training, Diffusion and Image Generation, Evaluating Models and MLOps and Serving.

The maths you need — and how much

This guide’s framing You need three areas of mathematics, at the level of a first-year university course, and mostly the ability to read them rather than prove theorems. Here is an honest summary.

If that list worries you, good news: you can learn it alongside the track. The page The Maths You Need covers exactly this ground gently, and the free book Mathematics for Machine Learning goes deeper. Many people find the maths finally makes sense only once they have seen it in code.

A single neuron, learning

Here is the whole idea of deep learning at the scale of one unit. A neuron takes some inputs, multiplies each by a weight, adds a bias, and squashes the result through an activation function. Training means nudging the weights so the output moves towards the answer we want. Install NumPy and run this:

import numpy as np

x = np.array([0.5, -1.2, 3.0])    # three inputs
w = np.array([0.8, 0.1, -0.4])    # one weight per input
b = 0.2                           # bias
target = 1.0                      # the answer we want

def forward(w, b):
    z = w @ x + b                 # weighted sum
    return 1 / (1 + np.exp(-z))   # sigmoid squashes it into (0, 1)

for step in range(5):
    y = forward(w, b)
    loss = (y - target) ** 2
    grad_z = 2 * (y - target) * y * (1 - y)   # chain rule
    w = w - 0.5 * grad_z * x                  # nudge every weight
    b = b - 0.5 * grad_z
    print(f"step {step}: output {y:.3f}  loss {loss:.3f}")

The output starts at 0.327 and climbs to 0.887 in five steps, while the loss falls from 0.452 to 0.013. What happened:

• w @ x + b is a dot product: 0.8×0.5 + 0.1×(−1.2) + (−0.4)×3.0 + 0.2 = −0.72. A real layer does thousands of these at once as one matrix multiplication.
• The sigmoid turns −0.72 into 0.327. Without a non-linear step like this, stacking layers would add nothing: many linear maps in a row are still one linear map.
• The loss is the squared distance from the target.
• grad_z is the chain rule in one line: how the loss changes with the output (2(y − target)) times how the output changes with z (the sigmoid’s derivative, y(1 − y)). Multiplying by x gives how the loss changes with each weight.
• The update steps every weight a little against its gradient, scaled by a learning rate of 0.5. That is gradient descent.

A large language model is this, repeated across billions of weights, arranged in a cleverer architecture, with the gradient computed automatically by backpropagation. Nothing in it is more mysterious than these fifteen lines — there is just vastly more of it, which is where the genuinely hard questions begin.

Play with the learning rate: too small and progress crawls; too large and the ball overshoots and diverges. The same trade-off governs training a model with billions of parameters.

A study path and the classic texts

There is no single right order, but this sequence works for many self-taught engineers. Tap each step.

When you are done, The Expert’s Road takes you to current research.

Key terms on this page

Sources & further reading

• David Rumelhart, Geoffrey Hinton & Ronald Williams, “Learning representations by back-propagating errors”, Nature 323 (1986).
• Ian Goodfellow, Yoshua Bengio & Aaron Courville, Deep Learning (MIT Press, 2016).
• Christopher M. Bishop, Pattern Recognition and Machine Learning (Springer, 2006); Christopher M. Bishop & Hugh Bishop, Deep Learning: Foundations and Concepts (Springer, 2024).
• Aston Zhang, Zachary C. Lipton, Mu Li & Alexander J. Smola, Dive into Deep Learning (Cambridge University Press, 2023).
• Marc Peter Deisenroth, A. Aldo Faisal & Cheng Soon Ong, Mathematics for Machine Learning (Cambridge University Press, 2020).
• Andrej Karpathy, Neural Networks: Zero to Hero (video course, 2022 onwards).