Chapter 1 · Main chapter
What Is an LLM?
Before we write a single line of model code, the whole idea — where these things come from, how they learn, and the plan for building one yourself.
Chapter 1 is pure concept — no code. This page distils it into four scroll-driven acts; the actual building starts in chapter 2.
Act 1 · What is an LLM?
Artificial intelligence
Machine learning
Deep learning
Start wide. Artificial intelligence is any system that does something we'd call intelligent — understanding language, spotting patterns, making decisions.
That last box — the LLM — isn't a new kind of network. It's a specific architecture, borrowed from a 2017 breakthrough in machine translation.
Act 2 · The transformer, split in two
input · English
output · German
LLMs run on the transformer, introduced in 2017 for machine translation. It has two halves: an encoder that reads the input and a decoder that writes the output.
So GPT is the decoder half of a transformer, generating one word at a time. But where does the skill come from? The answer is a training task so simple it's easy to underestimate.
Act 3 · Learning to predict the next word
Here's the whole training task, and it's almost absurdly simple: given some words, predict the next one. That's it. That's what GPT learns to do.
The large in "large language model" is the other half of the story: it's not just the training text but the model. GPT-3 is 175 billion parameters across 96 transformer layers (§1.5) — and that scale buys breadth. One such model translates, summarizes, answers questions, and writes code (§1.2), which is why a single general LLM can replace a drawer full of task-specific ones.
Concept, architecture, training task — that's the what and the how. The rest of this site is the build, and it comes in three stages.
The rest of this site is one long build in three stages. Stage 1: build the LLM. Turn text into tensors, code attention, assemble the GPT architecture.
Four things to remember
- An LLM is a deep neural network. Zoom out and it nests neatly: AI ⊃ machine learning ⊃ deep learning ⊃ LLMs. It learns language from data rather than from hand-written rules.
- It runs on the transformer. GPT keeps the decoder half and generates text; BERT keeps the encoder half and classifies it. This site builds the GPT side.
- Its entire training task is predicting the next word. That's self-supervised — every sentence labels itself — which is exactly why it scales to hundreds of billions of tokens of raw text.
- You'll build it in three stages: code the architecture, pretrain it into a foundation model, then fine-tune it. One page per stage, starting with chapter 2.
Adapted from Build a Large Language Model (From Scratch) by Sebastian Raschka — chapter 1 (Apache 2.0). This is a conceptual overview; the interactive, code-backed chapters begin at Text to Tensors.