Direct Preference Optimisation (DPO) and Beyond

RLHF works, but it is complex: train a reward model, then run unstable reinforcement learning with four large models in memory. In 2023, Rafailov, Sharma, Mitchell and colleagues showed that the same objective can be optimised directly on preference data with a simple loss. Direct Preference Optimisation (DPO) quickly became one of the most widely used alignment methods, especially for open models.

Starting point: the RLHF objective#

This KL-regularised problem has a known closed-form optimal policy:

where $Z(x)$ is a (intractable) normalising constant.

The key insight: rearrange for the reward#

Solve for $r$:

Every reward function corresponds to a policy, and vice versa — "your language model is secretly a reward model". Now substitute this into the Bradley–Terry preference model. The troublesome $Z(x)$ appears in both rewards and cancels in their difference:

The DPO loss#

Maximise the likelihood of the observed preferences directly with respect to the policy parameters:

It looks like logistic regression: increase the (reference-relative) log-probability of the chosen response and decrease that of the rejected one. No sampling during training, no reward model, no value network — just forward passes of the policy and a frozen reference model on fixed preference pairs.

The gradient's intuition#

with implicit rewards $\hat{r}_\theta = \beta\log\frac{\pi_\theta}{\pi_{\text{ref}}}$. Examples where the model currently ranks the pair wrongly receive larger weight — like a focusing mechanism.

Implementation#

import torch
import torch.nn.functional as F

def sequence_logprob(model, input_ids, labels):
    """Sum of log-probs of response tokens (labels == -100 elsewhere)."""
    logits = model(input_ids).logits[:, :-1]
    tgt = labels[:, 1:]
    logp = torch.gather(logits.log_softmax(-1), 2, tgt.clamp(min=0).unsqueeze(-1)).squeeze(-1)
    return (logp * (tgt != -100)).sum(-1)

def dpo_loss(pi_chosen, pi_rejected, ref_chosen, ref_rejected, beta=0.1):
    """Inputs: summed log-probs of chosen/rejected responses under policy and frozen reference."""
    logits = beta * ((pi_chosen - ref_chosen) - (pi_rejected - ref_rejected))
    loss = -F.logsigmoid(logits).mean()
    reward_acc = (logits > 0).float().mean()           # how often implicit reward ranks correctly
    return loss, reward_acc

print(dpo_loss(torch.tensor([-12.0, -30.0]), torch.tensor([-15.0, -28.0]),
               torch.tensor([-13.0, -29.0]), torch.tensor([-14.0, -29.0])))

In practice, libraries such as TRL provide DPOTrainer; data is a set of (prompt, chosen, rejected) triples, typically starting from an SFT model that is also used as the frozen reference.

Hyperparameters and practice#

• $\beta$ (typically 0.01–0.5) controls deviation from the reference: smaller $\beta$ allows larger changes.
• Low learning rates (e.g. $5\times10^{-7}$ to $5\times10^{-6}$ for full fine-tuning), one to a few epochs.
• Preference data can be human labelled, AI labelled, or constructed by sampling several responses from the current model and ranking them (on-policy data tends to work better).
• Monitor implicit reward accuracy, reward margins, and — crucially — generation quality on held-out prompts.

Known issues and variants#

• Likelihood displacement: DPO can decrease the absolute probability of both chosen and rejected responses while increasing their gap, sometimes harming quality.
• Overfitting to preference data and length exploitation (preferring longer responses) — regularisation or length-controlled variants help.
• Offline: DPO learns from a fixed dataset; iterative/online DPO regenerates preference pairs with the current model.

DPO vs RLHF#

Both remain in use; many modern pipelines combine SFT, preference optimisation (DPO-style) and RL with verifiable rewards.