Chapter 7 // Architectures

Convolutional Neural Networks

An MLP flattens an image into a vector and forgets that pixels have neighbors. Convolutions keep the grid — sliding one small detector everywhere — and in doing so buy translation invariance, locality, and a hundredfold cut in parameters.

Adapted from the original chapter of Dive into Deep Learning · CC BY-SA 4.0 · the Python shown is the book's; the figures run a faithful TypeScript port in your browser

Every network so far treated an image as a flat list of numbers. Permute the pixels and the model can't tell — which means it never used the one fact that matters most about an image: nearby pixels are related. A one-megapixel photo into a thousand hidden units is already a billion weights. Convolutional neural networks fix both problems at once, by baking two priors — locality and translation invariance — straight into the architecture.

Act 1 · A kernel slides over an image

01/05

48 pixels, one small detector, slid everywhere

A fully connected layer flattens this image to 48 numbers and forgets which pixels touch. A convolution keeps the grid and slides one small kernel across it — the same detector everywhere, the essence of translation invariance.

That sliding-window sum is the whole primitive. The output is slightly smaller than the input on every pass, and for one axis its size is:

n_\textrm{out} = \left\lfloor \frac{n - k + 2p + s}{s} \right\rfloor

where $k$ is the kernel size, $p$ the padding per side, and $s$ the stride. Those two knobs — padding and stride — are how you control the shape of what comes out.

Act 2 · Controlling the map — padding, stride, pooling

01/05

8 − 3 + 1 = 6 — the border pixels fall off

Every convolution shrinks the map: a 3×3 kernel turns 8×8 into 6×6. Stack ten 5×5 layers on a 240² image and you shave 30% off the edges — the corner pixels are barely used at all.

So far, one grid in, one grid out. Real images arrive as a stack of channels, and the layers between them carry many more. This is where a CNN keeps its capacity.

Act 3 · Channels — a stack of feature detectors

01/05

each channel gets its own kernel plane

Real images have channels — red, green, blue. A kernel grows to match: one plane per input channel. Each plane cross-correlates its own channel.

Cross-correlation, padding, stride, channels, and pooling — that is the entire toolbox. Stack it in the right order and you get the network that started it all.

Act 4 · LeNet-5 — the first working CNN

01/05

the MLP flattened this to 784 numbers — LeNet keeps the grid

Now assemble the parts. Chapter 5 flattened this 28×28 digit into a 784-vector and lost its shape. LeNet, from 1989, keeps the grid and processes it with convolutions instead.

The lab

Three things to drive yourself: what different kernels do to a real image (and how one can be learned), how padding and stride set the output size, and the small translation invariance that pooling buys.

▸ lab

Kernels: pick one, or learn one

Apply a hand-designed filter, then watch gradient descent discover the edge kernel by itself.

kernel

The same 3×3 window, different weights: edges, blur, sharpen, or emboss — all from one sliding kernel.

▸ lab

Padding, stride, and the output size

One formula predicts every output shape. Drag the knobs and watch it hold.

input n8

kernel k3

padding p1

stride s1

⌊(8 − 3 + 2·1 + 1) / 1⌋

output 8×8

Output matches input: this is “same” convolution (p = (k−1)/2, s = 1).

▸ lab

Pooling, and the invariance it buys

Toggle max vs average, then slide the bright pixel and see when the pooled map stops caring.

mode

shift pixel →+0

invariant ✓ — same pooled map as the unshifted pixel

Inside one 2×2 window, where the pixel sits doesn't matter — both max and average report the same thing. Shift far enough to leave the window and it finally moves.

Cross-correlation, or convolution?

The real code

Every act above carries a </> chip with its section's exact PyTorch. The whole chapter rests on one tiny nested loop:

conv-layer.md (PyTorch)

def corr2d(X, K):  #@save
    """Compute 2D cross-correlation."""
    h, w = K.shape
    Y = torch.zeros((X.shape[0] - h + 1, X.shape[1] - w + 1))
    for i in range(Y.shape[0]):
        for j in range(Y.shape[1]):
            Y[i, j] = (X[i:i + h, j:j + w] * K).sum()
    return Y

LeNet-5, end to end (the full network)

lenet.md (PyTorch)

def init_cnn(module):  #@save
    if type(module) == nn.Linear or type(module) == nn.Conv2d:
        nn.init.xavier_uniform_(module.weight)
 
class LeNet(d2l.Classifier):  #@save
    def __init__(self, lr=0.1, num_classes=10):
        super().__init__()
        self.save_hyperparameters()
        self.net = nn.Sequential(
            nn.LazyConv2d(6, kernel_size=5, padding=2), nn.Sigmoid(),
            nn.AvgPool2d(kernel_size=2, stride=2),
            nn.LazyConv2d(16, kernel_size=5), nn.Sigmoid(),
            nn.AvgPool2d(kernel_size=2, stride=2),
            nn.Flatten(),
            nn.LazyLinear(120), nn.Sigmoid(),
            nn.LazyLinear(84), nn.Sigmoid(),
            nn.LazyLinear(num_classes))
 
# trained on Fashion-MNIST with cross-entropy + minibatch SGD
trainer = d2l.Trainer(max_epochs=10, num_gpus=1)
data = d2l.FashionMNIST(batch_size=128)
model = LeNet(lr=0.1)
model.apply_init([next(iter(data.get_dataloader(True)))[0]], init_cnn)
trainer.fit(model, data)

[from the book]

Everything above runs a TypeScript port of these operations, verified against a Python implementation of the same maths (npm run verify:cnn — 26 checks to 1e-6, including a finite-difference proof that the learned-kernel gradient is real calculus). Honest deviations, all so the run stays checkable and the figures stay legible: the book trains LeNet-5 on 60,000 Fashion-MNIST photos and lands in the mid-eighties; training a CNN in the browser during scroll is out of scope, so the LeNet act animates the exact layer-by-layer shape flow the book prints with layer_summary (61,706 parameters in all) rather than a live run. The "learn a kernel" lab runs the book's own gradient-descent loop — squared error, plain SGD step — and it converges to ≈ [1, −1] in ten steps (loss 6.13 → 0.015); only the random initialization uses this project's PRNG so the run reproduces. Every other figure is the verified operation on the book's own worked examples (the 19/25/37/43 cross-correlation, the 56 two-channel sum, the 4/5/7/8 max-pool). The book goes further than four acts can — the receptive- field theory, the full multi-channel cost analysis, and the training curves are all worth reading in the original.

Three things to remember

A convolution is a small kernel slid over a grid. It multiplies and sums under a window at every position, so the same detector runs everywhere — translation invariance — while only ever looking at a local patch. That is how a CNN uses a few hundred weights where an MLP needed billions.
Padding, stride, channels, and pooling shape the map. Padding preserves size, stride downsamples, pooling coarsens and adds a little shift-invariance, and channels are where the capacity lives — deeper layers trade spatial resolution for channel depth.
LeNet is the recipe, assembled. Two conv-and-pool blocks turn one channel into sixteen while the map shrinks 28² → 5², then a small dense tail reads out ten classes — the first working CNN, and the blueprint for everything after.

Adapted from Chapter 7 of Dive into Deep Learning by Zhang, Lipton, Li, and Smola (CC BY-SA 4.0). The figures run this site's verified TypeScript port; all computation happens live in your browser.