An interactive companion to the book
Build a Large Language Model, visually.
Every chapter of Sebastian Raschka's LLMs from Scratch rebuilt as an explorable page — the algorithms run live in your browser, so you can poke at them while you read.
Chapters
Chapter 2 — Working with Text Data
Tokenization & Data Loading
Turning raw text into the token-ID tensors an LLM trains on: a naive word tokenizer and its <|unk|> problem, the sliding-window data loader, and the embedding tables that make IDs into vectors.
Byte Pair Encoding from Scratch
The tokenizer behind GPT-2, GPT-4, and Llama 3 in three scroll-driven acts — watch merges happen, grow a real vocabulary, then tokenize anything against OpenAI’s actual GPT-2 vocab.
Lookup Is a Matrix Multiply
The bonus that demystifies embeddings: an embedding lookup is exactly a matrix multiply on a one-hot vector — the layer just skips the multiply-by-zeros and indexes the row instead.
Same Tokens, Different Speed
Four byte pair encoding implementations — tiktoken, Hugging Face, OpenAI’s original, and the one you built — produce identical token IDs but run about 10× apart in speed.
The Data Loader, on Plain Numbers
The GPT sliding-window data loader run over 0, 1, 2, … 1000 instead of tokens, so the next-token target is obviously just the next number.
Chapter 3 — Attention Mechanisms
Attention from Scratch
The engine of the transformer, built in four passes on one tiny sentence: a weightless weighted average, trainable query/key/value projections, causal masking, and multi-head attention — every score and softmax running live.
Same Attention, Different Speed
Nine ways to write multi-head attention — stacked heads, one big weight split, einsum, PyTorch's fused kernel — all return the identical output but run up to ~2× apart on CPU and GPU.
Chapter 4 — Implementing a GPT Model
Implementing a GPT Model
Assemble GPT-2 from its parts — layer norm, GELU feed-forward, shortcut connections, and the transformer block — into a 163M-parameter model. Count every weight and watch the untrained model generate live.
The KV Cache
Why generation without a cache re-computes the same keys and values every step — and how a growing KV cache turns that quadratic waste into one new column per token.
Chapter 6 — Fine-tuning for Classification
Fine-tuning GPT for Spam
Repurpose the pretrained GPT-2 as a spam detector: balance the SMS dataset, swap the 50257-way head for a 2-way one, unfreeze just the last block, and fine-tune to ~96% test accuracy — every curve and verdict from the real executed run.
LoRA: Low-Rank Adaptation
Fine-tune the same classifier by freezing all 124M weights and learning tiny low-rank adapters instead — same accuracy for 2.7M trainable parameters. The decomposition, the zero-init trick, and a live parameter calculator.
What Actually Matters in Fine-tuning
Nineteen variants of the spam classifier — last vs first token, how many layers to unfreeze, model size, LoRA, padding — measured against the baseline. The design decisions, quantified.
Chapter 7 — Fine-tuning to Follow Instructions
Fine-tuning to Follow Instructions
Turn a pretrained GPT into an instruction follower: the Alpaca prompt template, a custom collate function that pads and masks targets with -100, and LLM-as-a-judge evaluation — the collate math running live.
Preference Tuning with DPO
Direct Preference Optimization aligns the instruction-tuned model to preferred over dispreferred responses — no reward model, no reinforcement learning, just one loss: −logσ(β·margin), running live.