Fine-tuning to Follow Instructions

Once every example is a single formatted string, it's tokenized and batched. But batching needs equal lengths — and that's where the one genuinely new piece of code lives.

Those batches feed ordinary PyTorch DataLoaders — batch size 8, moved to the GPU when there is one — over an 85/5/10 split of the 1,100 pairs: 935 training, 55 validation, 110 test (§7.4).

With the data prepared, fine-tuning is the same training loop as pretraining, just on this new data and a larger model — GPT-2 medium (355M), since 124M is too small to follow instructions well.

The model clearly learned to obey — but "how well?" has no string-match answer.

The lab

Type any instruction and watch it become the exact string the model trains on, with a live GPT-2 token count.

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Rewrite the sentence in a more formal tone.

### Input:
hey whats up

Then build a batch yourself and drive the collate function: toggle the −100 padding mask and the optional instruction mask (exercise 7.2), on real token ids.

The real code

The whole prompt format is one function — a fixed preamble, then labelled sections, with the input block dropped when empty:

def format_input(entry):
    instruction_text = (
        f"Below is an instruction that describes a task. "
        f"Write a response that appropriately completes the request."
        f"\n\n### Instruction:\n{entry['instruction']}"
    )
    input_text = f"\n\n### Input:\n{entry['input']}" if entry["input"] else ""
    return instruction_text + input_text

The collate function pads a batch, builds shifted targets, and masks all but the first padding token to −100:

def custom_collate_fn(batch, pad_token_id=50256, ignore_index=-100,
                      allowed_max_length=None, device="cpu"):
    batch_max_length = max(len(item) + 1 for item in batch)
    inputs_lst, targets_lst = [], []
    for item in batch:
        new_item = item + [pad_token_id]                       # one end-of-text token
        padded = new_item + [pad_token_id] * (batch_max_length - len(new_item))
        inputs  = torch.tensor(padded[:-1])                    # drop the trailing pad
        targets = torch.tensor(padded[1:])                     # shift right by one
 
        mask = targets == pad_token_id                         # replace all but the
        indices = torch.nonzero(mask).squeeze()                # first pad token with
        if indices.numel() > 1:                                # ignore_index (-100)
            targets[indices[1:]] = ignore_index
        if allowed_max_length is not None:
            inputs, targets = inputs[:allowed_max_length], targets[:allowed_max_length]
        inputs_lst.append(inputs); targets_lst.append(targets)
    return torch.stack(inputs_lst).to(device), torch.stack(targets_lst).to(device)

# Same train_model_simple loop as pretraining — just a bigger model and this data.
optimizer = torch.optim.AdamW(model.parameters(), lr=0.00005, weight_decay=0.1)
train_losses, val_losses, tokens_seen = train_model_simple(
    model, train_loader, val_loader, optimizer, device,
    num_epochs=2, eval_freq=5, eval_iter=5,
    start_context=format_input(val_data[0]), tokenizer=tokenizer)
 
# Evaluate free-text responses with a larger model (Llama 3 via Ollama):
def generate_model_scores(json_data, json_key, model="llama3"):
    scores = []
    for entry in json_data:
        prompt = (f"Given the input `{format_input(entry)}` and correct output "
                  f"`{entry['output']}`, score the model response `{entry[json_key]}`"
                  f" on a scale from 0 to 100, where 100 is the best score. "
                  f"Respond with the integer number only.")
        scores.append(int(query_model(prompt, model)))
    return scores

Four things to remember

1. One template, every task. Wrapping every example in the same Alpaca prompt teaches the model the shape of an instruction — and it must match at inference.
2. Padding is bookkeeping, not signal. The collate function pads to a common length, shifts targets by one, and masks padding to −100 so the loss ignores it — keeping just the first end-of-text token as a learnable "stop".
3. Fine-tuning reuses everything. The training loop is pretraining's, unchanged; only the data and the model size differ.
4. Open-ended output needs an open-ended judge. The accuracy metric from classification fine-tuning can't grade free text, so a larger model scores it — our 355M model averages ~50/100, and preference tuning can refine it further.

Adapted from Build a Large Language Model (From Scratch) by Sebastian Raschka — original notebook (Apache 2.0). The prompt formatting and collate tensors run a TypeScript port verified against the Python; the model responses, loss curve, and Llama-3 judge scores are the its real GPT-2-medium outputs, shown as attributed data.