Chapter 13 // Training at Scale

Computational Performance

A good design can be an order of magnitude faster than a bad one — the difference between training in a week and missing the deadline. This chapter is the systems layer under everything you've built: how code becomes a graph, how work is queued, what the silicon can actually do, and how to spread a batch across many GPUs.

Adapted from the original chapter of Dive into Deep Learning · CC BY-SA 4.0 · the Python shown is the book's; the timelines run this site's cost model with disclosed units, and the data-parallel run is a verified TypeScript port

Every previous chapter cared about what the network computes. This one cares about how fast, because at scale that decides whether an idea is trainable at all. We start where your Python meets the machine — with the surprising fact that, on a GPU, almost none of your code runs when you think it does.

Act 1 · Two ways to run a program

01/04

Imperative execution: Python runs fancy_func(1, 2, 3, 4) one statement at a time — e is 3, f is 7, g is 10 — bouncing back to the interpreter after every single operation, and storing every intermediate just in case.

Compilation removes the interpreter from the inside of a computation. The next idea removes it from between computations — and it is already switched on every time you touch a GPU.

Act 2 · The queue behind your code

01/04

When you run z = x * y + 2on a GPU, Python doesn't compute anything. Each statement drops a task into the backend's queueand returns immediately — the frontend is free at t = 6 while the C++ backend is still working. This is why the book's torch benchmark looks impossibly faster than NumPy.

That whole trade-off is a single inequality. Enqueue a task in $t_1$ , compute it in $t_2$ , return its result in $t_3$ . Waiting on every one of $n$ results costs

n \, (t_1 + t_2 + t_3),

while enqueueing them all and synchronizing once costs only

t_1 + n \, t_2 + t_3

as long as compute keeps up with the frontend ( $t_2 \ge t_1$ , so the backend never starves). The enqueue and fetch overheads are paid once instead of $n$ times — which is exactly why you synchronize per minibatch, not per operation.

That queue has a second payoff: if the backend can reorder independent work, it can also run it on different devices at once.

Act 3 · Free parallelism

01/03

Two GPUs, two independent workloads — but a synchronize between them forces the devices to take turns: 10 kernels each, 20 ticks total. The barrier, not the hardware, is the bottleneck.

Every speedup so far — folding, queueing, overlapping — is really about not letting an expensive resource sit idle. To know which resource is expensive, you have to know the machine.

Act 4 · The iron underneath

01/05

This is the machine your model actually runs on. Training shuffles bytes between RAM, CPU, GPU, disk and network — and each pipe has a different width, from 500 GB/son the GPU's own memory to 10 GB/s of high-end Ethernet. Whichever pipe you saturate first is your speed limit.

The latency ladder is also the argument for more hardware: when one GPU is the bottleneck, add GPUs. But a network is a single object, so how do you split it?

Act 5 · One batch, many GPUs

01/05

Three ways to spread training over k GPUs: cut the network between layers, cut every layer across its channels — both drown in synchronization — or give every GPU the whole model and split the data. The book, and practice, pick door three.

Data parallelism turns training into an allreduce after every minibatch. On a real cluster, how those gradients travel is the whole game.

Act 6 · The 160-megabyte question

01/05

The book's worked example: a 4-GPU server, 160 MB of gradients on each card after every minibatch, 16 GB/s per PCIe link. Same math as the last act — but now the route the bytes take decides how long training waits.

The lab

Three things from the acts, yours to drive: the async pipeline and where it stops helping, the eight-orders-of-magnitude latency ladder, and the ring allreduce whose cost refuses to grow with the number of GPUs.

▸ lab

Drive the pipeline

Drag the three costs — find where async stops helping, and what it does to the queue.

synchronize after every task — 56 ticks

one final barrier — 28 ticks

n·(t₁+t₂+t₃) = 56 vs t₁ + n·t₂ + t₃ = 28 · speedup ×2.00

tasks n = 8enqueue t₁ = 2compute t₂ = 3fetch t₃ = 2barrier every — (end only)

Queue peaks at 4 pending tasks. Barriers cap memory; each one also pays t₁+t₃ again.

▸ lab

How long is a nanosecond?

Pick a reference — every other latency rescales as if the reference took one second.

if L1 cache hit (1.5 ns) took one second…

action	actual	feels like
L1 cache hit	1.5 ns	1 s (ref)
Float add / multiply / FMA	1.5 ns	1.0 s
L2 cache hit	5 ns	3.3 s
Branch mispredict	6 ns	4.0 s
L3 cache hit (unshared)	16 ns	11 s
Mutex lock/unlock	25 ns	17 s
GPU shared memory access	30 ns	20 s
Memory ref. (local CPU, 64 MB)	46 ns	31 s
Memory ref. (remote socket, 64 MB)	70 ns	47 s
GPU global memory access	200 ns	2 min
Intel Optane random read	305 ns	3 min
Send 4 KB over 100 Gbps fabric	1 µs	11 min
Compress 1 KB (Google Snappy)	3 µs	33 min
Launch a CUDA kernel	10 µs	2 h
Send 4 KB over 10 Gbps Ethernet	10 µs	2 h
Transfer 1 MB over NVLink	30 µs	6 h
Transfer 1 MB over PCIe	80 µs	15 h
Read 4 KB randomly (NVMe SSD)	120 µs	22 h
Read 1 MB sequentially (NVMe SSD)	208 µs	2 days
Read 4 KB randomly (SATA SSD)	500 µs	4 days
Round trip in the same datacenter	500 µs	4 days
Read 1 MB sequentially (disk)	5 ms	39 days
HDD random access (seek + rotation)	10 ms	77 days
Packet CA → Netherlands → CA	150 ms	3.2 years

The table is the book's (after Eliot Eshelman). With L1 as the reference, a random disk seek takes ~2.5 months and a transatlantic packet ~3 years — now click HDD random access and watch the fast rows collapse to nothing. Every performance trick in this chapter is a way to stay near the top of this table.

▸ lab

Walk the ring

Step the allreduce by hand — watch partial sums travel, and the total time refuse to grow with n.

start — GPU i holds its own gradient (value i+1) in every chunk · target: every chunk = 10

GPUs in the ring n = 4

steps: 2(n−1) = 6

per link per step: 160 MB / 4 = 40 MB

naive (all → GPU 0): 60 ms

ring, same links: 15 ms

Click step until the grid fills. Scatter builds each column's sum in one place; gather walks the finished chunks around.

The real code

Every act above carries a </> chip with its section's exact PyTorch — scripting a model, benchmarking the async backend, dropping a barrier to parallelize two GPUs, and the from-scratch train_batch with split_batch + allreduce:

multiple-gpus.md (PyTorch)

def allreduce(data):
    for i in range(1, len(data)):
        data[0][:] += data[i].to(data[0].device)   # sum onto GPU 0
    for i in range(1, len(data)):
        data[i][:] = data[0].to(data[i].device)     # broadcast back
 
def train_batch(X, y, device_params, devices, lr):
    X_shards, y_shards = split_batch(X, y, devices)
    ls = [loss(lenet(X_shard, device_W), y_shard).sum()
          for X_shard, y_shard, device_W in zip(
              X_shards, y_shards, device_params)]
    for l in ls:                 # backward on each GPU independently
        l.backward()
    with torch.no_grad():        # sum + broadcast every gradient
        for i in range(len(device_params[0])):
            allreduce([device_params[c][i].grad for c in range(len(devices))])
    for param in device_params:  # identical SGD step on every replica
        d2l.sgd(param, lr, X.shape[0])

The concise version — one line does all of it

multiple-gpus-concise.md (PyTorch)

net = nn.DataParallel(net, device_ids=devices)   # scatter + gather + allreduce
trainer = torch.optim.SGD(net.parameters(), lr)
loss = nn.CrossEntropyLoss()
for X, y in train_iter:
    trainer.zero_grad()
    X, y = X.to(devices[0]), y.to(devices[0])
    l = loss(net(X), y)
    l.backward()                 # gradients allreduced automatically
    trainer.step()
# scale the batch k-fold and nudge lr up when you add GPUs

[from the book]

This is a systems chapter, so the site draws a hard line the book's wall-clock benchmarks can't: no figure fakes a measurement. The timelines run a TypeScript port of the book's own cost model with the tick durations shown on screen; the quoted milliseconds (60 / 80 / 15 ms, the ≈6 ms ring) are the book's worked example, recomputed from its stated 160 MB gradients and 16 GB/s links. Only the data-parallel act runs a real training loop: chapter 5's identical verified XOR MLP, now trained with split_batch + allreduce + d2l.sgd, which is how the page can show the 1-GPU and 2-GPU runs landing on the same parameters to one part in 10¹⁵. Everything is checked by npm run verify:perf (the async/overlap schedules, the hardware arithmetic, the sync costs, the ring, and that k-GPU lockstep). The book goes further than six acts can — the full hardware tour, the parameter-server derivation, and multi-machine fault tolerance are all worth reading in the original.

Three things to remember

Your framework is lazier than it looks. PyTorch enqueues GPU work and returns immediately; compilation (torch.jit.script) folds the graph and drops the interpreter, while the async backend overlaps independent work for free. Blocking calls — print, .item(), .numpy() — end the party, so synchronize per minibatch, not per op.
Performance is a hardware story. Bandwidth and latency span eight orders of magnitude; the wins come from few, large, contiguous transfers that stay in cache and overlap with compute. Sketch the bottleneck on paper before you profile.
Data parallelism is allreduce, and routing is everything. Give every GPU the whole model, split the batch, sum the gradients — the math is unchanged. How you sum them (to one GPU, via the CPU, sharded, or around a ring) can swing a step from 80 ms to 6 ms.

Adapted from Chapter 13 of Dive into Deep Learning by Zhang, Lipton, Li, and Smola (CC BY-SA 4.0). The figures run this site's verified TypeScript port; all computation happens live in your browser.