Chapter 2 · Bonus material
Same Tokens, Different Speed
The from-scratch tokenizer isn't the only one. Four implementations of GPT-2's BPE agree on every token — but not on the clock.
Adapted from the original notebook by Sebastian Raschka · the Python shown is the book's; the demos run a faithful JavaScript port in your browser
Bonus · Same tokens, different speed
[15496, 11, 995, 13, 1148, 428, 438, 257, 1332, 30]
one sentence → 10 token IDs
Take one sentence — Hello, world. Is this-- a test? — and encode it with GPT-2's vocabulary. You get these 10 IDs.
The real code
Each library exposes the same encode call and returns the same IDs. From the comparison notebook:
text = "Hello, world. Is this-- a test?"
# 1. OpenAI's tiktoken (Rust core)
import tiktoken
tik = tiktoken.get_encoding("gpt2")
tik.encode(text) # [15496, 11, 995, 13, 1148, 428, 438, 257, 1332, 30]
# 2. OpenAI's original 2019 GPT-2 code
from bpe_openai_gpt2 import get_encoder
orig = get_encoder(model_name="gpt2_model", models_dir=".")
orig.encode(text) # [15496, 11, 995, 13, 1148, 428, 438, 257, 1332, 30]
# 3. Hugging Face
from transformers import GPT2TokenizerFast
hf = GPT2TokenizerFast.from_pretrained("gpt2")
hf(text)["input_ids"] # [15496, 11, 995, 13, 1148, 428, 438, 257, 1332, 30]
# 4. The from-scratch BPETokenizerSimple built in the last chapter
tokenizer_gpt2.encode(text) # [15496, 11, 995, 13, 1148, 428, 438, 257, 1332, 30]Two things to remember
- The standard is the vocabulary and merges, not the code. Any correct BPE implementation of GPT-2 returns identical token IDs.
- Speed is the differentiator. A from-scratch tokenizer is for understanding; tiktoken's compiled core is what you reach for in production — the same tokens, roughly 10× faster.
Adapted from Build a Large Language Model (From Scratch) by Sebastian Raschka — original notebook (Apache 2.0). The token IDs on this page are tiktoken's real output; the timings are its published benchmark.