Chapter 3 // Foundations

Linear Regression

One line, two knobs, and the algorithm that trains everything: watch gradient descent find the bottom of a loss surface, then drive it yourself.

Adapted from the original chapter of Dive into Deep Learning · CC BY-SA 4.0 · the Python shown is the book's; the figures run a faithful TypeScript port in your browser

Act 1 · A line meets data

01/05

Sixty examples, synthesized the way the book does it: a true rule (y = 2x + 4.2) plus noise. In real life you never see the rule — only these dots.

That "way that scales" has a name: gradient descent. To see it, stop looking at the line and start looking at the space of all lines.

Act 2 · Descending the loss surface

01/05

the line at the marker
w=-1.00 b=0.50
loss 9.43

Zoom out: every point on this map is a whole line — w across, b up, brightness = loss. Somewhere in the dark valley sits the optimum ◎. We start at the white-hot edge.

The loss whose slopes we just descended is squared error, averaged over the minibatch $\mathcal{B}$ , and the update is one rule applied over and over:

L(\mathbf{w},b)=\frac{1}{|\mathcal{B}|}\sum_{i\in\mathcal{B}}\frac{1}{2}\left(\mathbf{w}^\top\mathbf{x}^{(i)}+b-y^{(i)}\right)^{2},\qquad (\mathbf{w},b)\leftarrow(\mathbf{w},b)-\eta\,\partial_{(\mathbf{w},b)}L

Act 3 · The book's training run, replayed

01/04

loss per SGD step

epoch 0/3 · step 0/96

ŵ = 0.010 (true 2)

b̂ = 0.000 (true 4.2)

Now the real thing, exactly as the book runs it: 1,000 examples, minibatches of 32, learning rate 0.03, three epochs. The parameters start at ŵ ≈ 0, b̂ = 0 — a flat line, clueless.

The lab

Two claims from the acts you shouldn't take on faith: that loss-turning-knobs is something you could do by hand, and that SGD's settings genuinely matter.

▸ lab

Fit it yourself

Drag w and b. Least squares scores 0.480 — get within 0.005 of it.

w = -0.50b = 8.50

you are here, on the loss surface

loss 14.721

▸ lab

The training loop, in your hands

Pick a learning rate and batch size, hit play — then try to break it.

loss (log scale) · last 1 steps

batch

step 0 · ~epoch 0

ŵ = 0.014 (true 2) · b̂ = 0.000 (true 4.2)

loss 11.4439

The real code

Everything above runs a TypeScript port of the book's scratch implementation (verified against Python — see scripts/verify-linreg.sh). The book's own model is these few lines:

linear-regression-scratch.md (PyTorch)

class LinearRegressionScratch(d2l.Module):
    """The linear regression model implemented from scratch."""
    def __init__(self, num_inputs, lr, sigma=0.01):
        super().__init__()
        self.save_hyperparameters()
        self.w = d2l.normal(0, sigma, (num_inputs, 1), requires_grad=True)
        self.b = d2l.zeros(1, requires_grad=True)
 
    def forward(self, X):
        return d2l.matmul(X, self.w) + self.b
 
    def loss(self, y_hat, y):
        l = (y_hat - y) ** 2 / 2
        return d2l.reduce_mean(l)

The optimizer and the training run (full listing)

linear-regression-scratch.md (PyTorch)

class SGD(d2l.HyperParameters):
    """Minibatch stochastic gradient descent."""
    def __init__(self, params, lr):
        self.save_hyperparameters()
 
    def step(self):
        for param in self.params:
            param -= self.lr * param.grad
 
    def zero_grad(self):
        for param in self.params:
            if param.grad is not None:
                param.grad.zero_()
 
 
model = LinearRegressionScratch(2, lr=0.03)
data = d2l.SyntheticRegressionData(w=d2l.tensor([2, -3.4]), b=4.2)
trainer = d2l.Trainer(max_epochs=3)
trainer.fit(model, data)
 
with torch.no_grad():
    print(f'error in estimating w: {data.w - d2l.reshape(model.w, data.w.shape)}')
    print(f'error in estimating b: {data.b - model.b}')

Three things to remember

A model is knobs. Linear regression has two; GPT has billions. Loss turns any setting of the knobs into a single number to shrink.
The gradient is a compass, not a map. Descent never sees the whole surface — only the local slope, stepped against, over and over.
Minibatches buy speed with noise — and the noise is affordable. The shuffle–batch–step loop you just drove is the one that trains every model on this site.

Adapted from Chapter 3 of Dive into Deep Learning by Zhang, Lipton, Li, and Smola (CC BY-SA 4.0). The figures run this site's verified TypeScript port; all training happens live in your browser.