Convolutional Networks

Show a dense network a photo of a cat, then show it the same photo moved ten pixels to the left. To the network these are two unrelated lists of numbers: every pixel now meets a different weight. It must learn “cat in the top left”, “cat in the middle” and “cat in the bottom right” as separate facts, from separate examples. You would never design a vision system that way. A cat’s ear is an ear wherever it appears, and the detector for it should be the same everywhere. That one idea — reuse the same small detector at every position — is the convolutional network, and it carried computer vision from reading postcodes in the 1990s to beating the ImageNet benchmark in 2012.

Convolution is a sliding dot product

A convolution takes a small grid of weights called a kernel (or filter), places it over a patch of the image, multiplies the overlapping numbers and adds them up — a dot product, exactly like one neuron’s weighted sum from Neural Networks from Scratch. Then it slides one pixel along and does it again, and again, until it has covered the image. The grid of results is a feature map: a picture of where in the image the kernel’s pattern appears.

Here is a 6×6 “photo” whose left half is dark (0) and right half is bright (9), and a 3×3 kernel that subtracts the left column of each patch from the right column. Written out in NumPy with plain loops, so nothing is hidden:

import numpy as np

image = np.array([[0, 0, 0, 9, 9, 9],         # a 6x6 "photo": dark left, bright right
                  [0, 0, 0, 9, 9, 9],
                  [0, 0, 0, 9, 9, 9],
                  [0, 0, 0, 9, 9, 9],
                  [0, 0, 0, 9, 9, 9],
                  [0, 0, 0, 9, 9, 9]])
kernel = np.array([[-1, 0, 1],                # vertical-edge detector:
                   [-1, 0, 1],                # right column minus left column
                   [-1, 0, 1]])

def conv2d(img, k):
    kh, kw = k.shape
    out = np.zeros((img.shape[0] - kh + 1, img.shape[1] - kw + 1), dtype=int)
    for i in range(out.shape[0]):
        for j in range(out.shape[1]):
            patch = img[i:i + kh, j:j + kw]   # the 3x3 window under the kernel
            out[i, j] = np.sum(patch * k)     # dot product of window and kernel
    return out

print(conv2d(image, kernel))
print(conv2d(image, kernel.T))               # rotate the kernel: horizontal edges

[[ 0 27 27  0]
 [ 0 27 27  0]
 [ 0 27 27  0]
 [ 0 27 27  0]]
[[0 0 0 0]
 [0 0 0 0]
 [0 0 0 0]
 [0 0 0 0]]

Follow one window by hand. At the top-left position the 3×3 patch is all zeros, so the result is 0. One step right, the patch’s columns are 0, 0, 9: the kernel computes (−1×0 + 0×0 + 1×9) on each of three rows, giving 27. The next position covers 0, 9, 9 and also gives 27. At the far right the patch is all 9s; left and right columns cancel to 0. So the feature map lights up in a vertical stripe exactly where the image changes from dark to bright: the kernel is a vertical-edge detector. Transpose it and it looks for horizontal edges instead — and finds none, because this image has none.

Two small points. First, the output is 4×4, not 6×6: a 3×3 window fits in only four positions along each side. Second, mathematicians would call this cross-correlation; a true convolution flips the kernel first. Deep-learning libraries skip the flip, and since the kernel’s weights are learned, it makes no difference: the network simply learns the flipped version.

Classic image processing used hand-designed kernels like this one — the Sobel operator is a close relative. The insight of convolutional networks is to make the nine numbers learnable weights and let gradient descent discover whichever detectors the task needs.

Padding, stride, channels and pooling

Four settings control how a convolutional layer behaves. Tap each tile:

• 1PADPaddingAdd a border of zeros round the image before convolving. With a 3×3 kernel, one pixel of padding keeps the output the same size as the input, so edges are not lost and layers can be stacked without the image shrinking away.
• 2STRIDEStrideHow far the kernel jumps between positions. Stride 1 visits every position; stride 2 visits every other one, halving the height and width of the output. A cheap way to downsample.
• 3CHChannelsA colour image has 3 input channels (red, green, blue), so each kernel is really 3×3×3 and sums over all three. A layer has many kernels — say 64 — and each produces its own feature map, so the output has 64 channels. Deeper layers have more channels and smaller maps.
• 4POOLPoolingPooling shrinks a feature map by summarising each small block, usually by its maximum (max-pooling). A 2×2 max-pool halves height and width and keeps the strongest response in each block, which also makes the output slightly tolerant to small shifts.

The output size follows one formula, worth knowing by heart because shape errors are the commonest CNN bug:

For our 6×6 image and 3×3 kernel: no padding gives (6 − 3) ÷ 1 + 1 = 4; padding 1 gives (6 + 2 − 3) + 1 = 6; padding 1 with stride 2 gives ⌊5 ÷ 2⌋ + 1 = 3. PyTorch agrees:

import torch
from torch import nn
import torch.nn.functional as F

x = torch.zeros(1, 1, 6, 6)                   # (batch, channels, height, width)
x[..., 3:] = 9                                # same dark/bright image
k = torch.tensor([[-1., 0., 1.]] * 3).reshape(1, 1, 3, 3)   # (out_ch, in_ch, kh, kw)

print(F.conv2d(x, k)[0, 0])                   # PyTorch agrees with our loops
print("no padding:          ", tuple(F.conv2d(x, k).shape))
print("padding=1:           ", tuple(F.conv2d(x, k, padding=1).shape))
print("padding=1, stride=2: ", tuple(F.conv2d(x, k, padding=1, stride=2).shape))
print("2x2 max-pool:        ", tuple(F.max_pool2d(x, 2).shape))

conv = nn.Conv2d(3, 64, kernel_size=3, padding=1)        # RGB in, 64 feature maps out
n_conv = sum(p.numel() for p in conv.parameters())
pixels = 224 * 224
n_dense = (pixels * 3) * (pixels * 64) + pixels * 64     # dense layer, same input/output sizes
print(f"conv layer params:  {n_conv:,}")
print(f"dense layer params: {n_dense:,}")

tensor([[ 0., 27., 27.,  0.],
        [ 0., 27., 27.,  0.],
        [ 0., 27., 27.,  0.],
        [ 0., 27., 27.,  0.]])
no padding:           (1, 1, 4, 4)
padding=1:            (1, 1, 6, 6)
padding=1, stride=2:  (1, 1, 3, 3)
2x2 max-pool:         (1, 1, 3, 3)
conv layer params:  1,792
dense layer params: 483,388,358,656

PyTorch’s convolution reproduces our loop exactly. Note the tensor layout: images are (batch, channels, height, width), and kernels are (out_channels, in_channels, height, width). The last two lines of output are the subject of the next section.

Why not a dense layer?

The final calculation above is the case for convolution in two numbers. A convolutional layer that turns a 224×224 colour image into 64 feature maps of the same size has 1,792 parameters: 64 kernels of 3×3×3 weights plus one bias each, (3 × 3 × 3 + 1) × 64. A dense layer producing the same-sized output from the same input — every output connected to every input pixel — would need about 483 billion. That is not a model; it is a hard-disk problem.

Convolution gets away with so few parameters thanks to two assumptions about images, both true enough to be enormously useful:

• Locality. Pixels that matter to each other are usually near each other. An edge, a corner or a texture is defined by a small neighbourhood, so each output looks only at a 3×3 (or 5×5, or 7×7) patch.
• Weight sharing. A pattern worth detecting in one place is worth detecting everywhere, so the same kernel is reused at every position. As a result, if the input shifts, the feature map shifts with it — a property called translation equivariance.

But if each output sees only a 3×3 patch, how does the network ever recognise a whole face? By stacking. The receptive field of a unit is the patch of the original image that can influence it. One 3×3 layer: 3×3. Two stacked: 5×5. Three: 7×7. Pooling and stride make it grow much faster, because every later step covers twice as many original pixels. The network you will train below has a receptive field of 10×10 at its last convolutional stage — larger than the whole 8×8 digit. This is why the VGG networks of 2014 used stacks of small 3×3 kernels: two of them see as far as one 5×5 kernel with fewer weights and an extra non-linearity in between.

Stacking also builds a hierarchy. Peel the layers of a trained image network from the output inwards:

A small CNN on handwritten digits

Time to train one. The scikit-learn digits are tiny (8×8), so the network is tiny too: two convolutional layers of 3×3 kernels, each followed by ReLU and 2×2 max-pooling, then one dense layer to the ten classes. The only change to the data is shape: instead of flat rows of 64 numbers, each image becomes a (1, 8, 8) tensor with one channel. The training loop is the one from Training Well.

import torch
from torch import nn
from torch.utils.data import TensorDataset, DataLoader
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

torch.manual_seed(0)
X, y = load_digits(return_X_y=True)
X = X.reshape(-1, 1, 8, 8) / 16.0             # (N, channels=1, 8, 8) images, not flat rows
X_tr, X_va, y_tr, y_va = train_test_split(X, y, test_size=0.25, random_state=42)
X_tr, X_va = torch.tensor(X_tr, dtype=torch.float32), torch.tensor(X_va, dtype=torch.float32)
y_tr, y_va = torch.tensor(y_tr), torch.tensor(y_va)
loader = DataLoader(TensorDataset(X_tr, y_tr), batch_size=64, shuffle=True)

model = nn.Sequential(
    nn.Conv2d(1, 16, kernel_size=3, padding=1), nn.ReLU(), nn.MaxPool2d(2),   # -> 16 x 4 x 4
    nn.Conv2d(16, 32, kernel_size=3, padding=1), nn.ReLU(), nn.MaxPool2d(2),  # -> 32 x 2 x 2
    nn.Flatten(),                                                             # -> 128
    nn.Linear(32 * 2 * 2, 10))
print("parameters:", sum(p.numel() for p in model.parameters()))

opt, loss_fn = torch.optim.Adam(model.parameters(), lr=0.01), nn.CrossEntropyLoss()
for epoch in range(1, 9):
    model.train()
    for xb, yb in loader:
        opt.zero_grad()
        loss_fn(model(xb), yb).backward()
        opt.step()
    model.eval()
    with torch.no_grad():
        acc = (model(X_va).argmax(1) == y_va).float().mean().item()
    print(f"epoch {epoch}: val acc {acc:.3f}")

parameters: 6090
epoch 1: val acc 0.798
epoch 2: val acc 0.938
epoch 3: val acc 0.953
epoch 4: val acc 0.951
epoch 5: val acc 0.982
epoch 6: val acc 0.987
epoch 7: val acc 0.964
epoch 8: val acc 0.947

The shape comments trace the image through the network: 8×8 with one channel becomes 16 channels of 8×8 (padding keeps the size), pooled to 4×4; then 32 channels of 4×4, pooled to 2×2; flattened, that is 32 × 2 × 2 = 128 numbers for the final layer. The whole model has 6,090 parameters: (3×3×1 + 1) × 16 = 160 for the first convolution, (3×3×16 + 1) × 32 = 4,640 for the second and 128 × 10 + 10 = 1,290 for the classifier. It passes 98% validation accuracy by epoch 5 and peaks at 98.7% at epoch 6. The dip after that is the learning rate (0.01 is on the high side for Adam) and the small validation set: 450 images, so each image is worth 0.2 percentage points. Keeping the best checkpoint, as on the training page, would keep the epoch-6 weights.

On 8×8 digits a plain dense network does nearly as well; the images are so small that there is little spatial structure to exploit. The advantage shows up when you test something a dense network was never built for. Below, both kinds of network are trained the same way, then tested on the validation digits shifted one pixel to the right:

import torch
from torch import nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

X, y = load_digits(return_X_y=True)
X_tr, X_va, y_tr, y_va = train_test_split(X.reshape(-1, 1, 8, 8) / 16.0, y, test_size=0.25, random_state=42)
X_tr, X_va = torch.tensor(X_tr, dtype=torch.float32), torch.tensor(X_va, dtype=torch.float32)
y_tr, y_va = torch.tensor(y_tr), torch.tensor(y_va)
X_shift = torch.roll(X_va, shifts=1, dims=3)  # every test digit moved one pixel right
X_shift[..., 0] = 0                           # (blank the column that wrapped round)

def train(name, model, epochs=6):
    torch.manual_seed(0)
    for m in model.modules():                 # fresh, seeded weights
        if hasattr(m, "reset_parameters"): m.reset_parameters()
    opt, loss_fn = torch.optim.Adam(model.parameters(), lr=0.01), nn.CrossEntropyLoss()
    for epoch in range(epochs):
        perm = torch.randperm(len(X_tr))
        for i in range(0, len(X_tr), 64):
            idx = perm[i:i + 64]
            opt.zero_grad()
            loss_fn(model(X_tr[idx]), y_tr[idx]).backward()
            opt.step()
    model.eval()
    with torch.no_grad():
        acc = lambda X: (model(X).argmax(1) == y_va).float().mean().item()
        n = sum(p.numel() for p in model.parameters())
        print(f"{name} params {n:5d}  val acc {acc(X_va):.3f}  shifted {acc(X_shift):.3f}")

mlp = nn.Sequential(nn.Flatten(), nn.Linear(64, 128), nn.ReLU(), nn.Linear(128, 10))
cnn = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
                    nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
                    nn.Flatten(), nn.Linear(128, 10))
train("MLP", mlp); train("CNN", cnn)

MLP params  9610  val acc 0.973  shifted 0.482
CNN params  6090  val acc 0.978  shifted 0.660

On the normal validation set the two are level (97.3% against 97.8%), and the CNN does it with fewer parameters. Move every digit a single pixel and the dense network falls to 48%; the CNN keeps 66%. The convolutional layers shift their feature maps along with the input, and max-pooling absorbs part of the movement. It is not perfectly shift-proof — the final dense layer still cares where features are, and with only 2×2 positions left a one-pixel shift of an 8×8 image is a big move — which is why practitioners also augment training data with small shifts, flips and crops so the network learns the rest of the invariance from examples.

What the learned filters do

No one told this network to look for edges. Its sixteen first-layer kernels began as random numbers. Yet in large networks trained on natural photographs, the first-layer filters reliably become oriented edge, line and colour-contrast detectors, as Krizhevsky and colleagues showed for AlexNet. The later layers are harder to read, but visualisation techniques — finding the image patches that most excite a unit, or optimising an input image to excite it — reveal detectors for textures, then parts, then objects. Consensus The hierarchy from simple local features to complex global ones is one of the most reproducible findings in deep learning. Debated How far individual units correspond to human-meaningful concepts is less clear; many units respond to mixtures of unrelated things, a problem that the study of mechanistic interpretability tries to untangle.

From LeNet to ResNet

The history of these ideas is told on Backprop and the Connectionist Revival and The Deep Learning Revolution. Here is the engineering thread in five steps; tap each.

The last step deserves a closer look, because the same trick sits inside every Transformer. He and colleagues noticed something odd: a 56-layer plain network had higher training error than a 20-layer one. That is not overfitting — the deeper network simply could not be optimised, even though it could in principle copy the shallow one and set its extra layers to do nothing. Their fix was to make “doing nothing” the default. A residual block computes a correction and adds it to its own input:

import torch
from torch import nn

class ResidualBlock(nn.Module):
    def __init__(self, channels):
        super().__init__()
        self.body = nn.Sequential(
            nn.Conv2d(channels, channels, 3, padding=1), nn.BatchNorm2d(channels), nn.ReLU(),
            nn.Conv2d(channels, channels, 3, padding=1), nn.BatchNorm2d(channels))

    def forward(self, x):
        return torch.relu(x + self.body(x))   # the skip connection: add the input back

block = ResidualBlock(16)
x = torch.randn(8, 16, 32, 32)
print(tuple(block(x).shape))

(8, 16, 32, 32)

If F learns nothing useful, its weights can shrink to zero and the block passes x straight through. And during backpropagation, the “+ x” gives the gradient a direct path around every block — the derivative of x with respect to itself is 1 — so the signal reaching early layers no longer has to survive a hundred multiplications. That is the vanishing-gradient problem from RNNs, LSTMs and Sequence Models attacked in depth rather than in time. For the input and output to be added, they must have the same shape, which is why the block uses padding 1 and keeps the channel count; where a ResNet does downsample, the skip path gets its own 1×1 convolution with stride 2.

Beyond convolution: vision transformers

In 2020, Alexey Dosovitskiy and colleagues at Google cut images into 16×16-pixel patches, turned each patch into a vector, and fed the sequence of patches to a standard Transformer — no convolutions at all. Their Vision Transformer (ViT, published at ICLR 2021) matched or beat the best CNNs, but only when pretrained on very large datasets: without convolution’s built-in assumptions of locality and weight sharing, it has to learn them from data. Since then the two families have borrowed from each other: hierarchical transformers such as Swin add back local windows, and modernised CNNs such as ConvNeXt (2022) adopted transformer-era training recipes and closed much of the gap. This guide’s framing Large multimodal models mostly use transformer-based image encoders, while CNNs remain common where compute is tight or data is modest — phones, cameras, factory inspection, many medical-imaging systems As of 2026. The attention mechanism itself is the subject of Transformers in Depth.

Key terms on this page

Sources & further reading

• Yann LeCun, Léon Bottou, Yoshua Bengio & Patrick Haffner, “Gradient-Based Learning Applied to Document Recognition”, Proceedings of the IEEE 86(11) (1998).
• Alex Krizhevsky, Ilya Sutskever & Geoffrey E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks”, NeurIPS (2012).
• Kaiming He, Xiangyu Zhang, Shaoqing Ren & Jian Sun, “Deep Residual Learning for Image Recognition”, CVPR (2016).
• Kunihiko Fukushima, “Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position”, Biological Cybernetics 36 (1980).
• Matthew D. Zeiler & Rob Fergus, “Visualizing and Understanding Convolutional Networks”, ECCV (2014); Karen Simonyan & Andrew Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition”, ICLR (2015).
• Alexey Dosovitskiy et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale”, ICLR (2021).
• Vincent Dumoulin & Francesco Visin, “A guide to convolution arithmetic for deep learning” (2016).
• PyTorch documentation: torch.nn.Conv2d As of 2026; Stanford CS231n notes on convolutional networks.