Chapter 9 // Architectures

Recurrent Neural Networks

Everything so far took a fixed-size input and produced one answer. But language, audio, and time series arrive as sequences of arbitrary length. Give a network a hidden state that loops back on itself, and it can carry the past forward — one step at a time.

Adapted from the original chapter of Dive into Deep Learning · CC BY-SA 4.0 · the Python shown is the book's; the figures run a faithful TypeScript port in your browser

Up to now every input was a fixed-length vector: a flattened image, a row of features. Sequences break that assumption — a sentence, a melody, a stock chart have no fixed length, and the order is the whole point. The move that unlocks them is simple: instead of one prediction from one input, predict the next element from the ones before it.

Act 1 · Predicting the next step

01/04

τ = 4 · one-step MSE 0.050

A sequence: xₜ = sin(0.01·t) + noise, a thousand points in time. Unlike tabular data, each value depends on the ones before it — order is the whole point.

A window of the last few values is enough to see one step ahead, but errors snowball when you forecast further. Before we build a better memory, we need to turn the messiest sequence of all — text — into numbers.

Act 2 · Turning language into numbers

01/04

raw text

The Time Traveller (for so it will be convenie…

lowercase · letters only

the␣time␣traveller␣for␣so␣it␣will␣be␣convenien

Text is a sequence too. First clean it — lowercase, letters only — then split into characters, the atomic tokens. We tokenize The Time Machine at the character level to keep the vocabulary tiny.

Counting n-grams runs out of data almost immediately. The fix is a model that carries a running summary of everything it has read, instead of a table of memorized counts. That summary is a hidden state, and a network that updates it is recurrent:

\mathbf{H}_t = \tanh(\mathbf{X}_t \mathbf{W}_{xh} + \mathbf{H}_{t-1} \mathbf{W}_{hh} + \mathbf{b}_h), \qquad \mathbf{O}_t = \mathbf{H}_t \mathbf{W}_{hq} + \mathbf{b}_q

The single new term, $\mathbf{H}_{t-1} \mathbf{W}_{hh}$ , is the loop — this step's state depends on the last. Everything else is an ordinary layer.

Act 3 · A network with memory

01/04

The RNN adds one thing to an MLP: a hidden state that feeds back into itself. Hₜ = tanh(Xₜ W_xh + Hₜ₋₁ W_hh + b_h). That W_hhloop is the memory — this step's state depends on the last.

The same weights act at every step, so an RNN is really a very deep network whose depth is the length of the sequence. That depth is its power and its curse — training it means backpropagating through every time step.

Act 4 · Training through time

01/04

ep 00“the hhhhhhhhhhhhhhhhhhhhhhhh”

ep 10“the tilien and and and and a”

ep 40“the bubbles that flashed and”

ep 80“the bubbles that flashed and”

Train the character RNN on the passage and perplexity falls from 25 toward 1.3. The samples tell the story: from "the hhhh…" to spaces and real words, learned one character at a time.

The lab

Three claims from the acts, yours to poke at: that a linear model's forecasts decay once you chase the horizon, that a tiny trained RNN really does capture the statistics of its text, and that the recurrent weights sit on a knife's edge between vanishing and exploding gradients.

▸ lab

The autoregressive model

Change the window τ and the horizon k. How far ahead can a linear model really see?

window τ = 4

horizon k = 16

one-step MSE 0.050

At 16 steps the fed-back errors compound — the forecast decays to a constant, whatever τ you pick.

▸ lab

Sample from the character RNN

Type a prefix, set the temperature, and let the trained model continue it one character at a time.

the time tfaveller for so it will be convenient to speak

prefix

temperature0.5

Low temperature keeps sampling close to the model's favorites: mostly real words, little surprise.

▸ lab

Gradient flow through time

Scale the recurrent weights and watch the backpropagated gradient over 24 time steps.

weight scale×1.00

spectral radius ρ 1.75

‖g‖ after 24 steps 8.1e-2

regime vanish

ρ < 1: the gradient decays geometrically. Long-range signal is lost — the model can't learn distant dependencies.

Backpropagation through time, in detail

Act 4 showed what goes wrong; the book shows exactly why.

[the product that vanishes or explodes]

Unrolling the recurrence and applying the chain rule, the gradient of the loss with respect to the hidden state at step $t$ works out to a sum of terms, each carrying a power of the recurrent weight matrix: $\partial L/\partial \mathbf{h}_t = \sum_{i=t}^{T} (\mathbf{W}_{hh}^\top)^{T-i}\,\mathbf{W}_{qh}^\top\,\partial L/\partial \mathbf{o}_{T+t-i}$ . Powers of a matrix are governed by its eigenvalues: any below 1 shrink to nothing, any above 1 blow up. That is the vanishing/exploding gradient problem in one line. Three strategies manage the cost: full backprop (exact but unstable and infeasibly long), regular truncation (detach the state every fixed number of steps — what everyone actually does, and a mild regularizer), and randomized truncation (unbiased but, in practice, no better). Clipping handles the explosions; the vanishing half waits for the gates of chapter 10.

The concise version

The from-scratch cell is instructive but slow. In practice you reach for the framework's optimized nn.RNN, which does the same recurrence in one call.

The real code

Every act above carries a </> chip with its section's exact PyTorch. The whole recurrent cell is four lines inside a loop over time:

rnn-scratch.md (PyTorch)

def forward(self, inputs, state=None):
    if state is None:
        state = torch.zeros((inputs.shape[1], self.num_hiddens))
    else:
        state, = state
    outputs = []
    for X in inputs:                      # loop over time steps
        state = torch.tanh(torch.matmul(X, self.W_xh) +
                           torch.matmul(state, self.W_hh) + self.b_h)
        outputs.append(state)
    return outputs, state

Training the character language model (the full loop)

rnn-scratch.md (PyTorch)

data = d2l.TimeMachine(batch_size=1024, num_steps=32)
rnn = RNNScratch(num_inputs=len(data.vocab), num_hiddens=32)
model = RNNLMScratch(rnn, vocab_size=len(data.vocab), lr=1)
# gradient clipping is applied between backward() and the optimizer step
trainer = d2l.Trainer(max_epochs=100, gradient_clip_val=1, num_gpus=1)
trainer.fit(model, data)
 
# generate 20 characters after a prefix
model.predict('it has', 20, data.vocab)

[from the book]

Everything above runs a TypeScript port of these operations, verified against a Python implementation of the same maths (npm run verify:rnn — 40 checks to 1e-6, plus a finite-difference proof that the manual backprop through time is real calculus). Honest deviations, all so the run stays checkable and the figures stay legible: the book trains a 32-hidden RNN on the full ~170,000-character Time Machine for 100 epochs; the site runs the identical algorithm — tanh cell, one-hot input, manual BPTT, gradient clipping, minibatch SGD, prefix decoding — on a short embedded excerpt so the whole run is reproducible and can be baked into the page (perplexity falls from ~25 toward ~1.3, and the samples really do converge on the passage's words). The autoregressive model of the first act is fit by least squares rather than the book's five epochs of SGD; both reach the same optimum. The vanishing/exploding demonstration multiplies powers of a real random recurrent matrix. The book goes further than four acts can — the full BPTT derivation, Laplace smoothing, randomized truncation, and the concise nn.RNN are all worth reading in the original.

Three things to remember

Sequences need memory, and memory is a hidden state. Rather than store a table of n-gram counts, an RNN keeps a running summary $\mathbf{H}_t$ and updates it with the same weights at every step — so the model size never grows with the sequence length.
An RNN is a very deep network in disguise. Unrolled through time it has one layer per step, trained by backpropagation through time — the ordinary chain rule, summed over every place the shared weights appear.
Depth in time is where gradients die. The backprop signal carries powers of the recurrent matrix, so it vanishes or explodes with its eigenvalues. Clipping survives the explosions; truncation limits the cost; and the vanishing problem is what the gated architectures of chapter 10 were built to solve.

Adapted from Chapter 9 of Dive into Deep Learning by Zhang, Lipton, Li, and Smola (CC BY-SA 4.0). The figures run this site's verified TypeScript port; all computation happens live in your browser.