Fine-tuning GPT for Spam

We have batches of padded token IDs and their labels. Now the model that reads them.

Freezing all but the top keeps the pretrained knowledge intact and makes training fast — only a few million weights move. Time to move them.

The lab

You against the model. Read a real held-out message, call it — then flip the card to see the fine-tuned GPT's verdict, its confidence, and the true label.

The real code

The demos run the numbers from the executed notebook. Here is the Python they mirror. First, the dataset — tokenize every message, then pad to the longest one with the <|endoftext|> token so a batch is a clean rectangle:

class SpamDataset(Dataset):
    def __init__(self, csv_file, tokenizer, max_length=None, pad_token_id=50256):
        self.data = pd.read_csv(csv_file)
        self.encoded_texts = [tokenizer.encode(t) for t in self.data["Text"]]
        if max_length is None:
            max_length = max(len(e) for e in self.encoded_texts)
        self.max_length = max_length
        # truncate, then right-pad every sequence to max_length
        self.encoded_texts = [
            e[:max_length] + [pad_token_id] * (max_length - len(e))
            for e in self.encoded_texts
        ]

The surgery is four steps: freeze everything, replace the head, unfreeze the last block and the final norm:

for param in model.parameters():          # 1. freeze all 124M weights
    param.requires_grad = False
 
torch.manual_seed(123)                     # 2. swap 768→50257 head for 768→2
model.out_head = torch.nn.Linear(in_features=768, out_features=2)
 
for param in model.trf_blocks[-1].parameters():  # 3. unfreeze last block
    param.requires_grad = True
for param in model.final_norm.parameters():      # 4. + final norm
    param.requires_grad = True

Classifying reads the last token's logits — the only position that has attended to the whole message:

def classify_review(text, model, tokenizer, device, max_length, pad_token_id=50256):
    ids = tokenizer.encode(text)[:max_length]
    ids += [pad_token_id] * (max_length - len(ids))
    x = torch.tensor(ids, device=device).unsqueeze(0)
    with torch.no_grad():
        logits = model(x)[:, -1, :]        # <-- last token only
    return "spam" if torch.argmax(logits, dim=-1).item() == 1 else "not spam"

def train_classifier_simple(model, train_loader, val_loader, optimizer, device,
                            num_epochs, eval_freq, eval_iter):
    train_losses, val_losses, train_accs, val_accs = [], [], [], []
    examples_seen, global_step = 0, -1
    for epoch in range(num_epochs):
        model.train()
        for input_batch, target_batch in train_loader:
            optimizer.zero_grad()
            loss = calc_loss_batch(input_batch, target_batch, model, device)
            loss.backward()
            optimizer.step()
            examples_seen += input_batch.shape[0]
            global_step += 1
            if global_step % eval_freq == 0:
                train_loss, val_loss = evaluate_model(
                    model, train_loader, val_loader, device, eval_iter)
                train_losses.append(train_loss)
                val_losses.append(val_loss)
        train_accs.append(calc_accuracy_loader(train_loader, model, device, num_batches=eval_iter))
        val_accs.append(calc_accuracy_loader(val_loader, model, device, num_batches=eval_iter))
    return train_losses, val_losses, train_accs, val_accs, examples_seen
 
# optimizer = AdamW(model.parameters(), lr=5e-5, weight_decay=0.1); num_epochs = 5

Four things to remember

1. Classification fine-tuning is surgery, not retraining. Freeze the pretrained body, replace the output head, and let only the top few layers adapt — a handful of the 124M weights actually move.
2. Balance beats size. A 6.5× ham/spam imbalance is undersampled to 747-per-class; a tiny, even dataset trains a strong classifier in minutes.
3. Classify from the last token. Causal attention means only the final position has read the whole message, so its output vector carries the verdict.
4. A little goes a long way. Five epochs on 1,045 training messages lifts a coin-flip head to ~96% held-out accuracy.

Adapted from Build a Large Language Model (From Scratch) by Sebastian Raschka — original notebook (Apache 2.0). The curves, accuracies, and inbox verdicts are torch's real outputs from fine-tuning GPT-2 (124M), reproduced with npm run build:spam and gated by npm run verify:spam.