Direct Preference Optimization: Your Language Model is Secretly a Reward Model

The problem with the RLHF pipeline

Standard RLHF aligns a language model in three stages: supervised fine-tuning (SFT), then training a separate reward model on human preference pairs, then optimizing the policy against that reward model with PPO. The last stage is the painful one. It keeps several models in memory at once (policy, reference, reward, and a value head), depends on an on-policy RL loop that is sensitive to hyperparameters, and is easy to destabilize — the policy can drift until it exploits flaws in the frozen reward model.

The DPO derivation

DPO starts from the same KL-regularized objective that RLHF optimizes, but instead of solving it with RL, it uses the closed-form solution. For the standard objective the optimal policy satisfies

$$\pi^*(y \mid x) = \frac{1}{Z(x)}\,\pi_\text{ref}(y \mid x)\,\exp\!\Big(\tfrac{1}{\beta}\, r(x, y)\Big),$$

which can be rearranged to express the reward in terms of the policy itself:

$$r(x, y) = \beta \log \frac{\pi(y \mid x)}{\pi_\text{ref}(y \mid x)} + \beta \log Z(x).$$

The intractable partition function $Z(x)$ cancels when you take a difference of rewards between two completions of the same prompt. Substituting into the Bradley–Terry preference model turns the whole problem into a simple binary classification loss on the policy.

The loss

$$L_\text{DPO} = -\,\mathbb{E}_{(x, y_w, y_l)} \left[\log \sigma\!\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_\text{ref}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_\text{ref}(y_l \mid x)}\right)\right]$$

The first term raises the log-probability of the preferred completion $y_w$; the second lowers it for the rejected completion $y_l$. The reference model and $\beta$ keep the policy from drifting too far — exactly the role the KL penalty plays in RLHF.

Tradeoffs vs PPO-based RLHF

DPO is dramatically simpler: one stage, no reward model, no sampling loop, stable supervised-style training. The costs are that it is purely offline (it only ever sees the fixed preference dataset, never fresh on-policy samples), it can over-fit to the preference data, and a strong explicit reward model still has uses — for best-of-n sampling, online exploration, and reward-signal reuse — that a DPO-trained policy does not directly provide.