Modern CNNs

Once you can wire a convolution to a pooling layer, the question becomes how to stack them. This chapter is a decade of answers, in the order they arrived — each a brief champion of the ImageNet competition, each keeping the good ideas of the last and adding one. Most of it is architecture: diagrams of blocks. But two of these ideas are real, checkable machinery, and they reshaped deep learning far beyond vision.

AlexNet proved the recipe scales, but it placed every layer by hand and hid most of its weight in two enormous fully connected layers. The next wave made the design modular — and went hunting for that fat.

Blocks made networks deeper, and depth exposed a new problem: the signal flowing through a deep stack drifts and destabilizes, and the gradient flowing back through it dies. The next two ideas are the fixes — and the reason this chapter matters most.

Batch normalization steadies the forward signal. But there is a second failure at depth — the backward one — and it needs a different, even simpler idea.

Residual connections and batch normalization together made hundred-layer networks routine. Everything after is variation: what you do with the skip, and who does the designing.

The lab

Three claims from the acts, yours to break: that batch norm makes a layer's output deaf to the scale of its input, that a residual skip is the difference between a gradient that survives depth and one that vanishes, and that every one of these architectures is just a traced stack of shapes and parameters.

The real code

Every act carries a </> chip with its section's exact PyTorch. The residual block — two idea-bearing lines of it — is the whole chapter in miniature:

class Residual(nn.Module):
    def __init__(self, num_channels, use_1x1conv=False, strides=1):
        super().__init__()
        self.conv1 = nn.LazyConv2d(num_channels, kernel_size=3, padding=1, stride=strides)
        self.conv2 = nn.LazyConv2d(num_channels, kernel_size=3, padding=1)
        self.conv3 = nn.LazyConv2d(num_channels, kernel_size=1, stride=strides) if use_1x1conv else None
        self.bn1 = nn.LazyBatchNorm2d()
        self.bn2 = nn.LazyBatchNorm2d()
 
    def forward(self, X):
        Y = F.relu(self.bn1(self.conv1(X)))
        Y = self.bn2(self.conv2(Y))
        if self.conv3:
            X = self.conv3(X)
        Y += X                 # the residual connection — everything hinges here
        return F.relu(Y)

def batch_norm(X, gamma, beta, moving_mean, moving_var, eps, momentum):
    if not torch.is_grad_enabled():
        # prediction: normalize with the running (moving) statistics
        X_hat = (X - moving_mean) / torch.sqrt(moving_var + eps)
    else:
        # training: statistics of THIS minibatch (dim 0 for FC, (0,2,3) for conv)
        mean = X.mean(dim=0)
        var = ((X - mean) ** 2).mean(dim=0)
        X_hat = (X - mean) / torch.sqrt(var + eps)
        moving_mean = (1.0 - momentum) * moving_mean + momentum * mean
        moving_var = (1.0 - momentum) * moving_var + momentum * var
    Y = gamma * X_hat + beta   # scale and shift
    return Y, moving_mean.data, moving_var.data

Three things to remember

1. Architectures are stacks of blocks, and the trend is efficiency. After AlexNet, progress meant composing repeatable blocks (VGG), replacing fat fully connected layers with 1×1 convs and global pooling (NiN), and running many kernel sizes in parallel (GoogLeNet) — often with fewer parameters, not more.
2. Batch normalization stabilizes the forward pass. Normalize each layer's activations to zero mean and unit variance over the minibatch, then let a learned scale and shift back in. It smooths optimization, permits bigger learning rates, and regularizes — and it behaves differently at test time, using running statistics.
3. Residual connections stabilize the backward pass. Adding the input back around a block means the gradient always has an identity path home, so it never vanishes — which is the only reason networks over 100 layers deep can be trained. Both ideas later became standard far outside vision.

Adapted from Chapter 8 of Dive into Deep Learning by Zhang, Lipton, Li, and Smola (CC BY-SA 4.0). The figures run this site's verified TypeScript port; all computation happens live in your browser.