Diffusion and Image Generation

Put a drop of ink in a glass of water and wait. The ink spreads until the water is evenly grey, and no amount of waiting will make it gather back into a drop. Physicists call that diffusion, and it only runs one way. In 2015 Jascha Sohl-Dickstein and colleagues asked a strange question: what if a neural network could learn to run it backwards — not for ink, but for images? Start with pure static, remove a little noise at a time, and end with a photograph that never existed. Five years later the idea was producing images that rivalled the best generators of the day, and it now sits underneath most of the image and video tools you can name.

Two processes: destroy, then rebuild

A diffusion model is defined by a pair of processes. The forward process is fixed and needs no learning: over T steps (1,000 in the original DDPM paper) it adds a little Gaussian noise to a clean example x₀ until, at step T, nothing recognisable is left. The reverse process is what we train: a network that, given a noisy example and the step number, removes a little of the noise. Generation is running the reverse process from fresh random noise.

Why go the long way round? Because each reverse step is a small, easy problem — “this is slightly noisy; make it slightly cleaner” — whereas a single leap from noise to a photograph is very hard to learn. Diffusion turns one impossible problem into a thousand manageable ones.

The forward process in NumPy

Each forward step multiplies the current value by √(1 − β_t) and adds Gaussian noise with variance β_t. A convenient property of Gaussians is that you do not have to loop through the steps: you can jump straight from the clean data to any step t in one line.

Let us apply it to an eight-number signal, using the linear schedule from the DDPM paper (β from 0.0001 to 0.02 over 1,000 steps) and one fixed draw of noise:

import numpy as np
rng = np.random.default_rng(42)

T = 1000
betas = np.linspace(1e-4, 0.02, T)          # DDPM's linear noise schedule
alpha_bar = np.cumprod(1 - betas)           # how much signal survives to step t

x0 = np.array([1.0, 1.0, 0.5, -0.5, -1.0, -1.0, -0.5, 0.5])   # a clean 1-D "signal"
eps = rng.normal(size=x0.shape)             # one fixed draw of Gaussian noise

for t in [0, 50, 200, 500, 999]:
    a = alpha_bar[t]
    xt = np.sqrt(a) * x0 + np.sqrt(1 - a) * eps     # jump straight to step t
    print(f"t={t:3d}  signal x{np.sqrt(a):.3f}  noise x{np.sqrt(1-a):.3f}  ", np.round(xt, 2))
print("noise   ", np.round(eps, 2))

t=  0  signal x1.000  noise x0.010   [ 1.    0.99  0.51 -0.49 -1.02 -1.01 -0.5   0.5 ]
t= 50  signal x0.985  noise x0.173   [ 1.04  0.8   0.62 -0.33 -1.32 -1.21 -0.47  0.44]
t=200  signal x0.810  noise x0.586   [ 0.99  0.2   0.85  0.15 -1.95 -1.57 -0.33  0.22]
t=500  signal x0.279  noise x0.960   [ 0.57 -0.72  0.86  0.76 -2.15 -1.53 -0.02 -0.16]
t=999  signal x0.006  noise x1.000   [ 0.31 -1.03  0.75  0.94 -1.96 -1.31  0.12 -0.31]
noise    [ 0.3  -1.04  0.75  0.94 -1.95 -1.3   0.13 -0.32]

Read down the columns. At step 0 the output is the signal. By step 200 the shape (high, high, middle, low, low, low…) is blurred but visible. By step 500 the noise dominates, and at step 999 the output is almost exactly the noise vector printed on the last line — the signal is multiplied by 0.006. The bars show how much of the original survives at each step:

• t = 0signal ×1.000
  Clean data.
• t = 50×0.985
  Lightly speckled; denoising here means fine detail.
• t = 200×0.810
  Shape visible through noise.
• t = 500×0.279
  Mostly noise; denoising here means deciding the overall layout.
• t = 999×0.006
  Indistinguishable from fresh noise.

That last observation is the whole trick. The end of the forward process is a distribution we can sample from trivially — rng.normal — so if we can learn to reverse it, we can generate. Notice also that different steps pose different problems: early in the reverse process the network decides composition; late, it polishes texture. Real image models see this division of labour clearly.

Learning to predict the noise

What should the network output? Ho, Jain and Abbeel (2020), in the paper that made the approach practical — Denoising Diffusion Probabilistic Models, or DDPM — found that a remarkably simple target works best in practice: give the network the noisy sample and the step, and ask it to predict the noise ε that was added. If you know the noise, you know the clean signal, because the forward formula can be rearranged.

This is ordinary supervised denoising regression — the same mean-squared error you used on Measuring Models — and that is a large part of why diffusion trains stably. There is no adversary and no balancing act; just “guess the noise”, millions of times, at every noise level. (The full DDPM derivation weights the error at each step from a variational bound; the simplified, unweighted version was found to give better samples.)

Sampling then runs the steps in reverse. At each step the network’s noise estimate is scaled and subtracted, and — except at the very last step — a little fresh noise is added back:

You can do the whole thing on a laptop in seconds. Our “dataset” is one-dimensional: numbers clustered near −2 and +2, a stand-in for two kinds of image. The denoiser is scikit-learn’s small neural network, given the noisy value and the step:

import numpy as np
from sklearn.neural_network import MLPRegressor
rng = np.random.default_rng(42)

T = 50
betas = np.linspace(1e-3, 0.2, T)           # a short schedule so the demo is fast
alphas = 1 - betas
alpha_bar = np.cumprod(alphas)

def data(n):                                # the "images": numbers near -2 or +2
    return rng.choice([-2.0, 2.0], size=n) + 0.2 * rng.normal(size=n)

# training set: noisy x_t, the step t, and the noise we added (the target)
n = 20000
x0, t = data(n), rng.integers(0, T, size=n)
eps = rng.normal(size=n)
xt = np.sqrt(alpha_bar[t]) * x0 + np.sqrt(1 - alpha_bar[t]) * eps
net = MLPRegressor(hidden_layer_sizes=(64, 64), max_iter=60, random_state=42)
net.fit(np.column_stack([xt, t / T]), eps)  # learn to predict the noise

# sampling: start from pure noise and denoise step by step
x = rng.normal(size=2000)
for step in reversed(range(T)):
    eps_hat = net.predict(np.column_stack([x, np.full_like(x, step / T)]))
    x = (x - betas[step] / np.sqrt(1 - alpha_bar[step]) * eps_hat) / np.sqrt(alphas[step])
    if step > 0:
        x += np.sqrt(betas[step]) * rng.normal(size=x.shape)

print("final alpha_bar:", round(alpha_bar[-1], 4))
print("samples near -2:", np.mean(np.abs(x + 2) < 0.6), " near +2:", np.mean(np.abs(x - 2) < 0.6))
print("histogram:", np.histogram(x, bins=[-4, -3, -1, 1, 3, 4])[0])

final alpha_bar: 0.0045
samples near -2: 0.493  near +2: 0.4995
histogram: [   0  993    3 1004    0]

Two thousand samples began as a single bell curve centred on zero. After fifty denoising steps, 99% of them sit within 0.6 of −2 or +2, split almost exactly half and half, and only 3 are stranded in the middle. The network never saw a rule saying “there are two clusters”; it learned, at every noise level, which way is “towards the data”. Points that started slightly negative were gently steered to −2, slightly positive to +2.

Everything in a real image model is a scaled-up version of this loop. The input is a tensor of millions of numbers instead of one; the denoiser is a U-Net (a convolutional network, see Convolutional Networks) or, increasingly, a Transformer; the step number is fed in as an embedding; and training runs on hundreds of millions of images.

Why it works, and why it is slow

Consensus Predicting the noise is closely related to estimating the score — the direction in which the probability of the data increases fastest. Song and Ermon (2019) built generators on exactly that idea, and later work showed the two views are the same family. Intuitively, the network learns a vector field that points from “noisy” towards “plausible”, at every level of blur, and sampling follows the arrows downhill from static.

The cost is speed. Our sampler called the network fifty times; DDPM called it a thousand. Each call on an image is a full forward pass of a large network, so the early diffusion models were painfully slow. Much research since has attacked the step count:

Latent diffusion, text and guidance

A 512 × 512 colour image has 786,432 numbers, most of which encode detail our eyes barely notice. Latent diffusion (Rombach, Blattmann, Lorenz, Esser and Ommer, 2022) first trains an autoencoder that compresses images into a much smaller grid — in the configuration popularised by Stable Diffusion, 64 × 64 × 4 = 16,384 numbers, 48 times fewer — and decodes them back. The diffusion model then works entirely in that compressed latent space; only at the end does the decoder turn the denoised latent into pixels. The paper’s models were released openly in 2022 as Stable Diffusion, which ran on a single consumer GPU and triggered an explosion of open tools.

To follow a prompt, the denoiser must see the text. A pretrained text encoder turns the prompt into a sequence of vectors, and the denoiser reads them through cross-attention layers at every step: each patch of the image asks “which words concern me?” (the same mechanism as in Transformers in Depth). Training pairs images with their captions, so the model learns to denoise towards images that match the words.

In practice, plain conditioning follows prompts loosely. Classifier-free guidance (Ho and Salimans, 2022) sharpens it with a neat trick. During training, the caption is randomly dropped for a fraction of examples, so a single network learns both a conditional prediction (with the prompt) and an unconditional one (with an empty prompt). At sampling time you compute both and extrapolate away from the unconditional one:

The bracket is “what the prompt adds”; multiplying it by w says “more of that, please”. It doubles the work per step, but it is one of the main reasons modern generators follow prompts as well as they do. The “negative prompt” in many tools replaces ∅ with text you want to steer away from.

Diffusion, GANs and autoregressive images

Diffusion was not the first way to generate images, and it may not be the last. A fair comparison:

Debated Which family will dominate is open. Diffusion-style models (often with Transformer denoisers) lead much of image and video generation today, but the lines are blurring, and companies rarely publish full architectures for their newest products As of 2026.

Whose images? The training data question

The skill of an image generator is learned from its training pairs, and for the big open models those came largely from images scraped from the web with their alt-text — the LAION-5B dataset (Schuhmann et al., 2022) lists about 5.85 billion such pairs. That raises questions that technology alone cannot settle:

• Consent and copyright. Artists, photographers and image libraries have sued image-generator companies in several countries, arguing that training on their work without permission or payment is infringement; developers argue it is lawful learning from public material. Outcomes so far differ by country and claim As of 2026.
• Memorisation. Generators usually make new images, but not always: Carlini et al. (2023) extracted over a thousand near-copies of training images, including photos of real people, from public diffusion models. Duplicated training images were far more likely to be memorised.
• Harmful content in the data. A 2023 Stanford Internet Observatory investigation found links to child sexual abuse material in LAION-5B; the dataset was withdrawn and later re-released after cleaning. Web scale makes careful curation hard, not optional.
• Misuse of the output. Deepfakes, non-consensual imagery and misinformation; partial defences include watermarks and content credentials.

Debated Whether training a model is “copying” or “learning” is the heart of the legal fight, and courts have not converged. The Practitioner page Images, Audio and Video covers using these tools responsibly; the planned Society page “Copyright and Creativity” will cover the law.

Key terms on this page

Sources & further reading

• Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan & Surya Ganguli, “Deep Unsupervised Learning using Nonequilibrium Thermodynamics” (ICML, 2015).
• Jonathan Ho, Ajay Jain & Pieter Abbeel, “Denoising Diffusion Probabilistic Models” (NeurIPS, 2020).
• Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser & Björn Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models” (CVPR, 2022).
• Jonathan Ho & Tim Salimans, “Classifier-Free Diffusion Guidance” (2022).
• Yang Song & Stefano Ermon, “Generative Modeling by Estimating Gradients of the Data Distribution” (NeurIPS, 2019); Jiaming Song, Chenlin Meng & Stefano Ermon, “Denoising Diffusion Implicit Models” (2020).
• Prafulla Dhariwal & Alex Nichol, “Diffusion Models Beat GANs on Image Synthesis” (NeurIPS, 2021); Aditya Ramesh et al., “Zero-Shot Text-to-Image Generation” (2021).
• Nicholas Carlini et al., “Extracting Training Data from Diffusion Models” (USENIX Security, 2023); Christoph Schuhmann et al., “LAION-5B: An open large-scale dataset for training next generation image-text models” (NeurIPS Datasets and Benchmarks, 2022).