← Back to all posts
DPOPreference LearningLLM Alignment

DPO and Its Descendants: When Direct Alignment Beats Doing RL

How direct preference optimization works, what it gives up, and how to evaluate newer variants without confusing novelty with evidence.

O Ohara University · · 16 min read

TL;DR

DPO converts the KL-constrained preference-learning objective into a supervised loss over chosen and rejected responses. It avoids online rollouts, an explicit reward model, and a critic. In exchange, it is limited by the coverage and quality of a static preference dataset.

The Loss

For prompt (x), preferred response (y_w), rejected response (y_l), reference policy (\pi_{ref}), and temperature (\beta), DPO optimizes

[ -\log\sigma\left(\beta\left[\log\frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)}-\log\frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)}\right]\right). ]

The reference ratios keep the policy near its starting distribution. Unlike PPO, no reward-model score or on-policy advantage is required. The derivation and its relationship to preference-based RL matter more than the short implementation.

import torch.nn.functional as F

def dpo_loss(chosen_logratio, rejected_logratio, beta=0.1):
    margin = beta * (chosen_logratio - rejected_logratio)
    return -F.logsigmoid(margin).mean()

What Descendants Try to Change

Later methods alter which tokens count, how candidates are ranked, how noisy pairs are filtered, or whether an alternative response is supplied as context. Evaluate each variant by its documented objective and evidence; similarly named methods can be unrelated, and preprint results should not be treated as settled consensus.

When DPO Wins

Use DPO when you have reliable preference pairs, limited inference capacity, and a target distribution covered by the data. Prefer online RL or RLVR when the policy must discover behaviors absent from the dataset and a dependable reward can be evaluated during generation. A hybrid pipeline can use DPO for initialization and online optimization afterward.

Failure Modes

Preference labels can encode verbosity bias, annotator inconsistency, or positional artifacts. Easy rejected responses produce little useful discrimination; mislabeled high-margin pairs can dominate. Report data provenance, tie handling, reference model, beta, truncation rules, and both capability and safety evaluations.

References

  1. Rafailov, R. et al. (2023). Direct Preference Optimization.
  2. Azar, M. et al. (2023). A General Theoretical Paradigm to Understand Learning from Human Preferences.
  3. Ethayarajh, K. et al. (2024). KTO: Model Alignment as Prospect Theoretic Optimization.