Skip to content
LLMs from Scratch

Chapter 6 · Bonus — Appendix E

LoRA: Low-Rank Adaptation

The last chapter unfroze one transformer block to fine-tune. LoRA freezes the entire model and learns a handful of skinny adapter matrices instead — same accuracy, a fraction of the trainable weights.

Adapted from the original notebook by Sebastian Raschka · the Python shown is the book's; the demos run a faithful JavaScript port in your browser

LoRA · Low-Rank Adaptation

W · 768 × 768 = 589,824 weights

one attention matrix. Full fine-tuning updates every cell — and there are dozens of these.

01 / 05

Fine-tuning updates weight matrices like this 768 × 768 attention weight — over half a million numbers each, and GPT-2 has dozens. Storing a full copy of the update per task is expensive.

The lab

Adapters cost r · (in + out) parameters per layer. Slide the rank and pick which layers get them to see the trainable-parameter budget move.

LoRA parameter calculator

Rank and layer choice vs trainable parameters — pure arithmetic on the GPT-2 config.

164
adapt which layers
2.67Mtrainable parameters2.14 % of GPT-2 · 47 × smaller than full
full fine-tune
124.44M
last block + norm
7.09M
this LoRA
2.67M

Each LoRA adapter on a Linear(in → out) adds r · (in + out) parameters. Appendix E uses r = 16 on every Linear layer → 2.67M. Larger r means more capacity and more parameters; α (the scaling) is tuned separately and doesn't change the count.

The real code

A LoRA layer is two matrices and a scale. A is Kaiming-initialized; B is zero, so A·B starts as the zero update and the model begins as the pretrained one:

appendix-E.ipynb · §E.4
class LoRALayer(torch.nn.Module):
    def __init__(self, in_dim, out_dim, rank, alpha):
        super().__init__()
        self.A = torch.nn.Parameter(torch.empty(in_dim, rank))
        torch.nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
        self.B = torch.nn.Parameter(torch.zeros(rank, out_dim))
        self.alpha, self.rank = alpha, rank
 
    def forward(self, x):
        return (self.alpha / self.rank) * (x @ self.A @ self.B)

LinearWithLoRA wraps a frozen Linear and adds the adapter's output to it:

appendix-E.ipynb · §E.4
class LinearWithLoRA(torch.nn.Module):
    def __init__(self, linear, rank, alpha):
        super().__init__()
        self.linear = linear
        self.lora = LoRALayer(linear.in_features, linear.out_features, rank, alpha)
 
    def forward(self, x):
        return self.linear(x) + self.lora(x)   # W·x + (α/r)·(x·A·B)

Then freeze the whole model and swap every Linear for the LoRA-wrapped version — now only the adapters carry gradients:

appendix-E.ipynb · §E.4
def replace_linear_with_lora(model, rank, alpha):
    for name, module in model.named_children():
        if isinstance(module, torch.nn.Linear):
            setattr(model, name, LinearWithLoRA(module, rank, alpha))
        else:
            replace_linear_with_lora(module, rank, alpha)
 
for param in model.parameters():
    param.requires_grad = False
replace_linear_with_lora(model, rank=16, alpha=16)   # ~2.67M trainable
# optimizer = AdamW(model.parameters(), lr=8e-4, weight_decay=0.1)

Three things to remember

  1. A weight update can be low-rank. Replace a full 768 × 768 update with A·B (rank r) and you store r · (in + out) numbers instead of in · out — here, 24× fewer per matrix.
  2. Zero-init keeps it safe. Starting B at zero means the adapter is invisible at step 0, so training never has to first undo a random perturbation of the pretrained weights.
  3. Comparable accuracy, about a third of the weights. LoRA reaches ~97% test accuracy — on par with the main chapter's last-block-plus-norm fine-tune (7.09M trainable) — while training just 2.7M parameters (~38% of that), so each fine-tuned task costs a tiny adapter to store rather than a full model copy.

Adapted from Build a Large Language Model (From Scratch) by Sebastian Raschka — Appendix E notebook (Apache 2.0). The LoRA run's curves and accuracy are torch's real outputs (npm run build:lora); the parameter math is gated by npm run verify:lora.