Skip to content
LLMs from Scratch

Chapter 7 · Bonus material

Preference Tuning with DPO

Instruction tuning teaches the model what to say. Direct Preference Optimization teaches it which way of saying it we prefer — with a single, elegant loss.

Adapted from the original notebook by Sebastian Raschka · the Python shown is the book's; the demos run a faithful JavaScript port in your browser

Bonus · Prefer this, not that

instruction · input: freind --> friend

Evaluate the following phrase by transforming it into the spelling given.

chosen (preferred)

The spelling of the given phrase "freind" is incorrect, the correct spelling is "friend".

rejected (dispreferred)

The spelling of the given phrase "freind" is flat out wrong, get it together, the correct spelling is "friend".

01 / 05

Instruction tuning trains one target answer. But correct answers differ in style — and we often prefer one. Here the same fix comes politely (chosen) or bluntly (rejected).

The lab

The loss depends only on how much more the policy prefers the chosen response over the rejected one, compared to the frozen reference. Drag the rewards and β and watch it move — this is the real compute_dpo_loss running live.

DPO loss explorer

Set how much the policy prefers each response over the reference, and β — the loss updates live.

margin

1.20

loss

0.635

vs ln 2

-0.058

Policy prefers the chosen response → loss below ln 2. Good.

lossln 2

reward margin (chosen − rejected) → · loss = −log σ(β · margin)

The real code

The DPO loss is the whole idea in five lines. It compares each model's preference for chosen over rejected, then squashes the difference through a log-sigmoid:

dpo-from-scratch.ipynb
def compute_dpo_loss(model_chosen_logprobs, model_rejected_logprobs,
                     reference_chosen_logprobs, reference_rejected_logprobs, beta=0.1):
    model_logratios = model_chosen_logprobs - model_rejected_logprobs
    reference_logratios = reference_chosen_logprobs - reference_rejected_logprobs
    logits = model_logratios - reference_logratios
 
    losses = -F.logsigmoid(beta * logits)                     # DPO, Eq. 7
 
    chosen_rewards = (model_chosen_logprobs - reference_chosen_logprobs).detach()
    rejected_rewards = (model_rejected_logprobs - reference_rejected_logprobs).detach()
    return losses.mean(), chosen_rewards.mean(), rejected_rewards.mean()

The log-probabilities come from a gather over the model's logits, averaged over just the response tokens (prompt and padding are masked out):

dpo-from-scratch.ipynb
def compute_logprobs(logits, labels, selection_mask=None):
    labels = labels[:, 1:].clone()                            # shift, like any LM target
    logits = logits[:, :-1, :]
    log_probs = F.log_softmax(logits, dim=-1)
    selected = torch.gather(log_probs, dim=-1, index=labels.unsqueeze(-1)).squeeze(-1)
    if selection_mask is not None:
        mask = selection_mask[:, 1:].clone()
        return (selected * mask).sum(-1) / mask.sum(-1)       # average over response tokens
    return selected.mean(-1)

Three things to remember

  1. DPO skips the reward model. RLHF trains a separate reward model and optimizes with RL; DPO derives an equivalent objective you can minimize with ordinary gradient descent on preference pairs.
  2. The reference keeps it honest. A frozen copy of the starting model anchors the loss, so β can bound how far the policy drifts — at margin 0 the loss is exactly ln 2.
  3. It tunes style, gently. DPO nudges tone and phrasing, not knowledge, and is prone to collapse — so it's trained for one epoch at a tiny learning rate.

Adapted from Build a Large Language Model (From Scratch) by Sebastian Raschka — original bonus material (Apache 2.0). The DPO loss, log-probabilities, and preference collate run a TypeScript port verified against the Python; the preference data, training curves, and model responses are its real executed outputs, shown as attributed data.