What Actually Matters in Fine-tuning

The lab

All nineteen runs, sortable. Click a column to rank by it — accuracy or training time — and read off the trade-offs yourself.

The real code

Every row is one command-line flag on the same script:

python additional_experiments.py                              # row 1 (baseline)
python additional_experiments.py --trainable_token_pos first  # row 2
python additional_experiments.py --trainable_layers last_two_blocks  # row 4
python additional_experiments.py --trainable_layers all       # row 5
python additional_experiments.py --model_size "gpt2-large (774M)"    # row 7
python additional_experiments.py --trainable_layers lora --lora_rank 16 --lora_alpha 16  # row 11
python additional_experiments.py --context_length "model_context_length"  # row 13
python additional_experiments.py --disable_causal_mask        # row 17

Three things to remember

1. Ablations turn choices into evidence. Change one knob, hold the rest, re-measure — the only honest way to know which decision mattered.
2. The causal mask dictates the token position. Classifying the last token isn't arbitrary; it's the only position with full context, and the 20-point gap versus the first token proves it.
3. More capacity has diminishing — sometimes negative — returns. Bigger models and more trainable layers don't monotonically help; the last two blocks of a 124M model are a sweet spot.

Adapted from Build a Large Language Model (From Scratch) by Sebastian Raschka — additional experiments (Apache 2.0). The results shown are its published A100 benchmarks, quoted as attributed data.