Multilayer Perceptrons

Every model so far has been linear: the output moves monotonically with each input. That is a strong assumption, and often a wrong one — no straight line separates a checkerboard, tells a cat from its photographic negative, or reads a pixel in the context of its neighbors. The fix is almost embarrassingly simple: add a layer of hidden units, and a nonlinearity between the layers.

One hidden layer, one nonlinearity — that is the whole recipe. In symbols, the hidden representation and the output are:

The nonlinearity $\sigma$ is load-bearing: drop it and the two layers collapse back into one. With it, a single hidden layer is already a universal approximator — given enough units, it can fit any function. Learning the right weights is the hard part, and for that we need to know how gradients flow.

Backprop is just the chain rule run in reverse — but that reversed product of matrices is also where deep networks go wrong, long before they get a chance to learn anything at all.

Careful initialization keeps a deep network trainable. The next problem is the opposite one: a network with millions of parameters can memorize its training set outright. One of the most effective cures is to inject noise on purpose.

The lab

Three claims from the acts, yours to poke at: that hidden width is what buys a bent boundary, that the choice of activation decides whether gradients survive depth, and that dropout is unbiased noise that leaves the expected output alone.

How deep networks generalize

An over-parametrized network can fit anything — including randomly shuffled labels. By the classical story, that should doom it to overfit. Yet the same networks, trained on real labels, generalize beautifully. Deep learning breaks the tidy picture from chapter 3, and the theory is still catching up.

A real dataset: house prices

The book closes the chapter by leaving the lab and entering a Kaggle competition — predicting house prices in Ames, Iowa from 80 mixed features. The model is almost beside the point; the workflow is the lesson.

The real code

Every act above carries a </> chip with its section's exact PyTorch. The from-scratch MLP is just softmax regression with one extra layer and a ReLU:

class MLPScratch(d2l.Classifier):
    def __init__(self, num_inputs, num_outputs, num_hiddens, lr, sigma=0.01):
        self.W1 = nn.Parameter(torch.randn(num_inputs, num_hiddens) * sigma)
        self.b1 = nn.Parameter(torch.zeros(num_hiddens))
        self.W2 = nn.Parameter(torch.randn(num_hiddens, num_outputs) * sigma)
        self.b2 = nn.Parameter(torch.zeros(num_outputs))
 
def relu(X):
    return torch.max(X, torch.zeros_like(X))
 
@d2l.add_to_class(MLPScratch)
def forward(self, X):
    X = X.reshape((-1, self.num_inputs))
    H = relu(torch.matmul(X, self.W1) + self.b1)   # hidden layer
    return torch.matmul(H, self.W2) + self.b2       # output layer

# concise: stack Linear + ReLU + Dropout with nn.Sequential
class DropoutMLP(d2l.Classifier):
    def __init__(self, num_outputs, num_hiddens_1, num_hiddens_2,
                 dropout_1, dropout_2, lr):
        super().__init__()
        self.save_hyperparameters()
        self.net = nn.Sequential(
            nn.Flatten(), nn.LazyLinear(num_hiddens_1), nn.ReLU(),
            nn.Dropout(dropout_1), nn.LazyLinear(num_hiddens_2), nn.ReLU(),
            nn.Dropout(dropout_2), nn.LazyLinear(num_outputs))
 
hparams = {'num_outputs': 10, 'num_hiddens_1': 256, 'num_hiddens_2': 256,
           'dropout_1': 0.5, 'dropout_2': 0.5, 'lr': 0.1}
model = DropoutMLP(**hparams)
data = d2l.FashionMNIST(batch_size=256)
trainer = d2l.Trainer(max_epochs=10)
trainer.fit(model, data)

Three things to remember

1. Depth plus a nonlinearity buys expressiveness. Two linear layers collapse into one; a ReLU between them lets the network fold space into piecewise-linear regions and bend a boundary a straight line could never reach.
2. Backprop is the chain rule in reverse, and it can misfire. The gradient is a product of one matrix per layer — too small and it vanishes, too large and it explodes. Xavier initialization, random and scaled, keeps it alive.
3. Capacity is cheap; generalization is the work. A deep net can fit anything, so the real tools — weight decay, early stopping, dropout — all fight overfitting, and dropout does it by training a fresh thinned network every step.

Adapted from Chapter 5 of Dive into Deep Learning by Zhang, Lipton, Li, and Smola (CC BY-SA 4.0). The figures run this site's verified TypeScript port; all computation happens live in your browser.