Chapter 2 · Bonus material
The Data Loader, on Plain Numbers
The sliding window is easiest to believe when the tokens are just numbers. Here the target is, literally, the next integer.
Adapted from the original notebook by Sebastian Raschka · the Python shown is the book's; the demos run a faithful JavaScript port in your browser
The sliding window, on plain numbers
The same data loader as above — but over 0, 1, 2, … 1000, where the next-token target is just the next number.
the stream · window 1 of 997
every target is its input + 1 — the model learns to predict the next number
The real code
It reuses the exact GPTDatasetV1 from the main chapter and changes one
line — instead of tokenizing text, it parses integers straight from the file:
# number-data.txt is just "0 1 2 3 4 … 1000"
token_ids = [int(i) for i in txt.strip().split()] # the only change
# everything below is the chapter's sliding window, unchanged:
for i in range(0, len(token_ids) - max_length, stride):
input_chunk = token_ids[i : i + max_length]
target_chunk = token_ids[i + 1 : i + max_length + 1]
self.input_ids.append(torch.tensor(input_chunk))
self.target_ids.append(torch.tensor(target_chunk))dataloader = create_dataloader_v1(
raw_text, batch_size=1, max_length=4, stride=1, shuffle=False)
next(iter(dataloader)) # [tensor([[0, 1, 2, 3]]), tensor([[1, 2, 3, 4]])]Two things to remember
- The target is the input shifted by one. On numbers, "predict the next token" is visibly "predict the next number".
max_lengthandstrideshape the windows. Widen the window to see longer contexts; shrink the stride to overlap them — the same knobs as the real data loader.
Adapted from Build a Large Language Model (From Scratch) by Sebastian Raschka — original notebook (Apache 2.0). The windows on this page are produced by a faithful TypeScript port of its data loader, verified against that Python.