Chapter 2 // Foundations

Preliminaries

Before you can train anything, five survival skills: store data as tensors, push it through linear algebra, read a function's slope, get gradients for free, and reason about what you can't predict.

Adapted from the original chapter of Dive into Deep Learning · CC BY-SA 4.0 · the Python shown is the book's; the figures run a faithful TypeScript port in your browser

Every model in this book runs on the same small toolkit. None of it is deep learning yet — it's the arithmetic underneath, the part you stop noticing once it's fluent. Here it is, live, one tool at a time.

Act 1 · Everything is a tensor

01/05

A tensor is the one data structure everything here runs on: a flat block of numbers plus a shape. This is arange(12) — twelve of them, in a row.

One data structure, one rule that surprises everyone. Next: the operations you run on tensors, and how each one reshapes them.

Act 2 · The algebra of shapes

01/05

Linear algebra is the grammar of deep learning. Start with a matrix: a 2×3 grid of numbers — two rows, three columns.

Those operations have slopes — and a slope is exactly what tells a model which way to adjust. That's calculus, and it needs only one idea.

Act 3 · A derivative is a local slope

01/05

Calculus answers one question training needs: at this point, which way does the function move? Take f(x) = 3x² − 4x and ask for its slope at x = 1.

f'(x) = \lim_{h \to 0} \frac{f(x+h) - f(x)}{h}

Computing one slope by hand is easy. Computing millions, through a deep network, is not — so we hand the chain rule to a machine.

Act 4 · Gradients for free

01/05

Doing calculus by hand doesn't scale to a million parameters. So we let the framework do it. Take x = [0, 1, 2, 3] and the target y = 2·xᵀx.

Tensors, shapes, slopes, gradients: all exact. The last tool is for everything that isn't — noise, sampling, the future.

Act 5 · The law of large numbers

01/05

Probability is the language of uncertainty. Flip one fair coin: heads or tails, and no model on earth can call a single flip. This one came up tails.

The lab

Three claims from the acts, yours to poke at: that broadcasting has a rule you can predict, that random noise averages into a stable number, and that the likeliest label still isn't the decision.

▸ lab

Does it broadcast?

Pick two shapes. Matching or size-1 dimensions stretch; anything else is an error.

A rows

A cols

B rows

B cols

(3, 1) + (1, 2) → (3, 2)

▸ lab

Watch chaos become a number

Press play and let the coin flip 10,000 times. Then bias it and watch the estimate find the new truth.

true P(heads) = 0.50

flips 1 · heads 1

estimate 1.0000

error 0.5000

▸ lab

The positive test

A test comes back positive. Are you sick? Drag the base rate, sensitivity, and specificity.

base rate P(sick) = 0.15%sensitivity P(+ | sick) = 100.0%specificity P(− | healthy) = 99.0%

P(sick | positive)

13.1%

of everyone who tests positive: sick vs false alarm

per 100,000 people:

· 150 sick test positive

· 999 healthy test positive

· 1,149 positives in all

A positive result, and you're still more likely healthy — false alarms swamp a rare condition. This is why the likeliest label isn't the decision.

The real code

Two of the chapter's seven sections have no act above: pandas (loading and cleaning messy real-world data) and the API-documentation section (dir and help — how you get unstuck). Preprocessing is the whole ingest pipeline in a few lines — read a CSV, one-hot the categories, fill the gaps, hand the result to a tensor:

pandas.md (PyTorch)

data = pd.read_csv(data_file)
inputs, targets = data.iloc[:, 0:2], data.iloc[:, 2]
inputs = pd.get_dummies(inputs, dummy_na=True)
inputs = inputs.fillna(inputs.mean())
 
X = torch.tensor(inputs.to_numpy(dtype=float))
y = torch.tensor(targets.to_numpy(dtype=float))

Every act above carries a </> chip with its section's exact PyTorch; the rest of the toolkit, in one place:

Tensors, linear algebra, autograd, and getting unstuck (the tour)

ndarray.md · linear-algebra.md · autograd.md · lookup-api.md (PyTorch)

# create, reshape, broadcast
X = torch.arange(12, dtype=torch.float32).reshape(3, 4)
torch.arange(3).reshape((3, 1)) + torch.arange(2).reshape((1, 2))
 
# reduce, multiply, measure
A = torch.arange(6, dtype=torch.float32).reshape(2, 3)
A.sum(axis=0), torch.mv(A, torch.arange(3, dtype=torch.float32))
torch.norm(torch.tensor([3.0, -4.0]))
 
# autograd: write the forward pass, get exact gradients
x = torch.arange(4.0, requires_grad=True)
y = 2 * torch.dot(x, x)
y.backward()
x.grad
 
# when you're stuck, read the source
help(torch.ones)
print(dir(torch.distributions))

Three things to remember

Everything is a tensor, and shape is the whole game. Linear algebra moves data between shapes — reduce an axis, multiply a matrix by a vector — and broadcasting stretches size-1 axes so you almost never write a loop.
Autograd turns calculus into bookkeeping. A derivative is just a local slope; write the forward pass and the graph hands back every gradient exactly, so you can forget the calculus you just learned.
Probability is how a model reasons under uncertainty. Estimates converge as data grows — the law of large numbers — which is why data works at all, why we trust a held-out test set, and why a positive test still isn't a diagnosis.

Adapted from Chapter 2 of Dive into Deep Learning by Zhang, Lipton, Li, and Smola (CC BY-SA 4.0). The figures run this site's verified TypeScript port; all computation happens live in your browser.