Text to Tensors

From here we hand the job to GPT-2's real byte pair encoder, which never meets a word it can't split. The story becomes one stream of 5,145 token IDs — ready to be sliced into training examples.

Those windows are still just integers. The last step turns each ID into something a network can actually learn from — a vector.

The lab

Three sliders and the real token stream: build the exact inputs and targets tensors the data loader hands the model, and step through them batch by batch.

And the naive tokenizer from Act 1, loaded live. Type anything — watch which words it knows, and how everything unfamiliar collapses to one <|unk|>.

The real code

The demos above run TypeScript ports; here is the Python they mirror, from the original notebook. The sliding-window data loader is the heart of the chapter — the target is the input shifted by one:

class GPTDatasetV1(Dataset):
    def __init__(self, txt, tokenizer, max_length, stride):
        self.input_ids, self.target_ids = [], []
        token_ids = tokenizer.encode(txt, allowed_special={"<|endoftext|>"})
 
        # Slide a window across the token stream, stepping by `stride`.
        for i in range(0, len(token_ids) - max_length, stride):
            input_chunk = token_ids[i : i + max_length]
            target_chunk = token_ids[i + 1 : i + max_length + 1]
            self.input_ids.append(torch.tensor(input_chunk))
            self.target_ids.append(torch.tensor(target_chunk))
 
    def __len__(self):
        return len(self.input_ids)
 
    def __getitem__(self, idx):
        return self.input_ids[idx], self.target_ids[idx]

Turning IDs into vectors is two embedding tables and a sum — token identity plus position:

token_embedding_layer = torch.nn.Embedding(vocab_size, output_dim)
pos_embedding_layer   = torch.nn.Embedding(context_length, output_dim)
 
token_embeddings = token_embedding_layer(inputs)              # (batch, length, dim)
pos_embeddings   = pos_embedding_layer(torch.arange(context_length))
input_embeddings = token_embeddings + pos_embeddings          # what the model reads

import re
 
 
class SimpleTokenizerV2:
    def __init__(self, vocab):
        self.str_to_int = vocab
        self.int_to_str = {i: s for s, i in vocab.items()}
 
    def encode(self, text):
        preprocessed = re.split(r'([,.:;?_!"()\']|--|\s)', text)
        preprocessed = [item.strip() for item in preprocessed if item.strip()]
        # Any word not in the fixed vocabulary becomes <|unk|>.
        preprocessed = [
            item if item in self.str_to_int else "<|unk|>" for item in preprocessed
        ]
        return [self.str_to_int[s] for s in preprocessed]
 
    def decode(self, ids):
        text = " ".join([self.int_to_str[i] for i in ids])
        return re.sub(r'\s+([,.:;?!"()\'])', r'\1', text)  # tidy spaces before punctuation
 
 
def create_dataloader_v1(txt, batch_size=4, max_length=256,
                         stride=128, shuffle=True, drop_last=True, num_workers=0):
    tokenizer = tiktoken.get_encoding("gpt2")
    dataset = GPTDatasetV1(txt, tokenizer, max_length, stride)
    return DataLoader(
        dataset, batch_size=batch_size, shuffle=shuffle,
        drop_last=drop_last, num_workers=num_workers,
    )

Three things to remember

1. Tokenizing is lookup, and a fixed vocabulary is brittle. The naive tokenizer must reserve <|unk|> for everything it never saw — which is why real models use byte pair encoding instead.
2. Training data is manufactured, not given. A sliding window over one long token stream produces every (input, target) pair; the target is just the input shifted one step right.
3. IDs carry identity; positions carry order. Token embeddings and positional embeddings are summed into the vectors the transformer actually reads.

Adapted from Build a Large Language Model (From Scratch) by Sebastian Raschka — original notebook (Apache 2.0). The token IDs and data-loader batches on this page are its real printed outputs; the token and positional embedding values are torch's seed-123 output — verified number-for-number against the Python even where the notebook itself doesn't print them (its positional example is 4×256).