Chapter 10 // Architectures

Modern RNNs

A plain RNN forgets: multiply enough Jacobians and the gradient vanishes long before it reaches the start of a sequence. The fix is a cell you can write to, keep, and read on command — and once sequences flow reliably, two RNNs can translate one language into another.

Adapted from the original chapter of Dive into Deep Learning · CC BY-SA 4.0 · the Python shown is the book's; the figures run a faithful TypeScript port in your browser

Chapter 9 built a recurrent network and then broke it: the gradient is a long product of matrices, and long products either vanish or explode. Gradient clipping tames the explosions; vanishing needs a deeper fix. The answer, from 1997 and still everywhere today, is to replace each recurrent unit with a gated memory cell — a value you can choose to keep untouched for as many steps as you like.

Act 1 · The memory cell

01/05

An LSTM replaces each plain recurrent unit with a memory cell guarded by three sigmoid gates. Every value lands in (0, 1): the input gate decides what to let in, the forget gate what to keep, the output gate what to reveal.

The three gates and the input node are four small fully connected layers, all reading the same two inputs — the current token and the previous hidden state:

\mathbf{I}_t = \sigma(\mathbf{X}_t\mathbf{W}_{xi} + \mathbf{H}_{t-1}\mathbf{W}_{hi} + \mathbf{b}_i), \quad \tilde{\mathbf{C}}_t = \tanh(\mathbf{X}_t\mathbf{W}_{xc} + \mathbf{H}_{t-1}\mathbf{W}_{hc} + \mathbf{b}_c)

The forget and output gates have the identical shape. Everything that makes an LSTM special is in how they combine: $\mathbf{C}_t = \mathbf{F}_t\odot\mathbf{C}_{t-1} + \mathbf{I}_t\odot\tilde{\mathbf{C}}_t$ . A leaner design drops one gate and merges the cell into the hidden state.

Act 2 · A lighter gate — GRU

01/05

The gated recurrent unit keeps the idea but trims the parts. Two gates, not three, and no separate cell: a reset gate R and an update gate Z.

One cell — LSTM or GRU — is a layer. Stack them, or run them in both directions, exactly as you would any other layer.

Act 3 · Deeper, and both ways

01/05

Start with what chapter 9 built: a single recurrent layer, unrolled across time. Each cell feeds the next along the sequence — deep in time, but shallow in representation.

Reliable sequence models unlock the payoff: mapping one sequence to a different one. Translation is the canonical case, and the architecture is two RNNs facing each other.

Act 4 · Translate — sequence to sequence

01/06

Machine translation is sequence-to-sequence: variable-length in, variable-length out, and no alignment between them. Each sentence is tokenized, capped with an ⟨eos⟩, and padded to a fixed width. Our toy stand-in: reverse the string.

Greedy decoding takes the single most likely token at each step. It is fast and usually good — but "usually" leaves room for a smarter search.

Act 5 · Beam search

01/05

The Translate act decoded greedily: take the single most likely token every step. Cheap, and usually fine — this toy path scores 0.048. But greedy is myopic: one locally best choice can foreclose a better sequence.

The lab

Three claims from the acts, yours to drive: that a near-1 forget gate is what lets a memory (and its gradient) survive across time; that widening the beam recovers a sequence greedy threw away; and that BLEU rewards fluent word order while punishing translations that are too short.

▸ lab

The constant-error carousel

Enter a value at step 0, then set the forget and input gates and watch the cell state hold it — or let it fade.

after 13 steps, the t0 value is scaled by forget^13 = 100%

forget gate F = 1.00input gate I = 0.20add a second input at t6

Forget ≈ 1: the memory rides the recurrent edge forward almost unchanged — the gradient does too.

▸ lab

Beam width, and why greedy loses

Widen the beam from 1 (greedy) upward and watch a better sequence surface — the one greedy walked past.

greedy · width 1

A <eos> · prob 0.3 · score -0.716

beam · width 1

A <eos> · prob 0.3 · score -0.716

Width 1 is greedy: one path, one argmax per step. It commits early and can never reconsider.

beam width k = 1length penalty α = 0.75

α = 0 ranks by raw log-probability (favors short sequences); α = 1 divides fully by length. The book uses 0.75.

▸ lab

BLEU, n-gram by n-gram

Pick a candidate translation against a fixed reference and watch the precisions and the brevity penalty build the score.

reference: the cat sat on the mat

candidate: the cat sat on a mat

BLEU (k=2) = 0.803

candidate

longest n-gram k = 2

Longer matching n-grams are weighted more heavily, so fluent order is rewarded.

The dataset behind translation

The real code

Every act above carries a </> chip with its section's exact PyTorch. The encoder is a bare embedding-plus-RNN; the whole idea fits in a few lines:

seq2seq.md (PyTorch)

class Seq2SeqEncoder(d2l.Encoder):
    def __init__(self, vocab_size, embed_size, num_hiddens, num_layers, dropout=0):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_size)
        self.rnn = d2l.GRU(embed_size, num_hiddens, num_layers, dropout)
 
    def forward(self, X, *args):
        embs = self.embedding(X.t().type(torch.int64))
        outputs, state = self.rnn(embs)   # state = the context for the decoder
        return outputs, state

The LSTM and GRU cells, from scratch (the gate equations above)

lstm.md · gru.md (PyTorch)

# LSTM: three gates + an input node feed one memory cell
I = torch.sigmoid(X @ W_xi + H @ W_hi + b_i)
F = torch.sigmoid(X @ W_xf + H @ W_hf + b_f)
O = torch.sigmoid(X @ W_xo + H @ W_ho + b_o)
C_tilde = torch.tanh(X @ W_xc + H @ W_hc + b_c)
C = F * C + I * C_tilde
H = O * torch.tanh(C)
 
# GRU: two gates, no separate cell state
Z = torch.sigmoid(X @ W_xz + H @ W_hz + b_z)
R = torch.sigmoid(X @ W_xr + H @ W_hr + b_r)
H_tilde = torch.tanh(X @ W_xh + (R * H) @ W_hh + b_h)
H = Z * H + (1 - Z) * H_tilde

[from the book]

Everything above runs a TypeScript port of these operations, verified against a Python implementation of the same maths (npm run verify:modernrnn — the gate, forward, BLEU, and beam-search maths to 1e-6, plus a finite-difference proof that the encoder–decoder's backpropagation-through-time is real calculus). Honest deviations, all so the run stays checkable and the figures legible: the LSTM/GRU/deep/bidirectional acts read their numbers off a single fixed cell with disclosed weights, rather than a full character language model on The Time Machine. The Translate act trains a real RNN encoder–decoder — embedding, GRU encoder, context vector, GRU decoder with the context concatenated at every step, masked cross-entropy, teacher forcing, Adam with gradient clipping — but on a toy task, reversing a short symbol string, instead of the 167k-pair French–English set, so the whole run is deterministic and its translations are verifiable. Beam search runs over the book's exact toy probability tables. The book goes further than five acts can — the full translation pipeline, LSTM variants, and the theory of exposure bias are all worth reading in the original.

Three things to remember

Gated cells beat vanishing gradients by addition, not multiplication. An LSTM's cell state is updated by a gated sum, so a value — and its gradient — can ride a near-weight-1 edge across many steps instead of being repeatedly multiplied toward zero. GRUs get most of the benefit with two gates and no separate cell.
Depth and direction are just more layers. Stack recurrent layers to build richer representations; run one forward and one backward to condition each step on the whole sequence — great for labeling, useless for forecasting.
Sequence-to-sequence is an encoder that compresses and a decoder that generates. The encoder packs the input into a context vector; the decoder emits one token at a time, trained by teacher forcing, decoded by beam search, and scored by BLEU.

Adapted from Chapter 10 of Dive into Deep Learning by Zhang, Lipton, Li, and Smola (CC BY-SA 4.0). The figures run this site's verified TypeScript port; all computation happens live in your browser.