Direct preference optimization (DPO) learns from the same preference data as RLHF, with no separate reward model and no reinforcement learning loop. Classic RLHF trains a reward model on ranked responses, then optimises the policy against that reward with a reinforcement learning algorithm, usually PPO, under a KL penalty that keeps it close to the starting model. DPO reparameterises the reward model so the optimal policy can be written in closed form. That turns alignment into a simple classification loss over fixed chosen/rejected pairs, with no separate reward model and no sampling from the model during fine-tuning. PPO is therefore the optimiser inside RLHF, not a third alternative: the real contrast is between reward-based methods that use an actor-critic algorithm such as PPO and reward-free methods such as DPO.
The trade is not free. Removing the reward model also removes the online loop that lets RLHF generate and score fresh responses during training, and that loop lets online RL explore new regions of output space.
How Is DPO Different from RLHF in Practice?
RLHF trains two different models: a reward model that learns which behaviour people prefer, and a policy that the reward model then guides. The reward model is usually a pre-trained language model whose final layer is replaced by a regression head, so it outputs one number per prompt and response instead of a distribution over tokens. During RL the policy generates responses, the reward model scores them, and PPO updates the policy, which means sampling from the model inside the training loop and training several language models, at significant computational cost.
DPO skips the intermediate step and optimises the final policy directly on the preference data. It treats alignment as a supervised learning problem over preference pairs, which is simpler to implement and train. A 2025 survey of DPO frames the motivation the same way: RLHF’s reliance on explicit reward modelling costs computational efficiency and training stability, and DPO sidesteps both by optimising on preferences directly.
The operational difference worth holding onto is when the data is collected. DPO and the other direct alignment algorithms train on a dataset fixed ahead of time, while PPO generates fresh completions during training and scores them with a reward model.
What Does the DPO Loss Actually Optimise, and Why Does It Not Need a Separate Reward Model?
The DPO loss is a maximum-likelihood objective for the observed preferences under a Bradley-Terry model, written in terms of the policy rather than a reward. The Bradley-Terry model depends only on the difference in reward between two completions, so when the reward is rewritten as a function of the optimal policy and the reference policy, the troublesome partition function cancels out. What remains is an objective over the policy alone:
L_DPO(πθ; π_ref) = −E[(x, y_w, y_l) ~ D] log σ( β·log(πθ(y_w|x) / π_ref(y_w|x))
− β·log(πθ(y_l|x) / π_ref(y_l|x)) )
Here y_w is the preferred response, y_l the dispreferred one, π_ref the reference model (by default the policy before training) and β controls the trade-off with drifting from the reference. In code it is a handful of lines: subtract the rejected log-probability from the chosen one for both policy and reference, take the difference, and apply a log-sigmoid scaled by β.
The quantity β·log(πθ(y|x)/π_ref(y|x)) acts as an implicit reward defined by the language model and the reference model, which is why the original paper is subtitled “your language model is secretly a reward model”. The gradient raises the likelihood of the chosen completion, lowers the likelihood of the rejected one, and weights each pair by how badly the implicit reward currently orders it. That per-example weight is what stops the model degenerating the way a naive probability-ratio objective does.
One property matters for everything that follows. The DPO loss depends only on the margin between the chosen and rejected log-ratios, not on their absolute values.
How Many Preference Pairs Does DPO Need, and How Clean Must They Be?
DPO needs one record per comparison: a prompt, a preferred completion and a dispreferred completion to the same prompt. It usually works on pairwise comparisons, although the derivation generalises to full rankings through the Plackett-Luce model. RLHF needs comparable ranking data to fit its reward model, plus a supply of prompts to generate fresh rollouts from during RL, and PPO can use prompt-only data beyond the preference set.
We would distrust any single number offered as a minimum pair count, because the answer depends on how far the target behaviour sits from the SFT model. For scale, one 2024 study of format bias trained offline DPO on 60K pairs and the first round of online iterative DPO on 20K. Treat those as one paper’s settings, not a requirement.
Ivison et al. (2024) separated preference data, learning algorithm, reward model and policy training prompts, and found better preference data gave the largest improvements, up to 8% in instruction following and truthfulness. Label noise costs measurable accuracy: in a 2025 study of adaptive batch-wise sample scheduling for DPO, noisy samples pulled DPO’s test accuracy on the HH dataset down to about 58% from 64% in the noise-free setting.
Two further properties of the data also matter:
- Distribution match with the starting model. Xu et al. (2024) found DPO performance is significantly affected by the distribution shift between the model’s own outputs and the preference dataset; with an Alpaca SFT model and SafeRLHF preference data, DPO reached only a 55.4% safety rate.
- Some on-policy completions. Including completions from the model being tuned helps the contrastive loss optimise the right part of token space.
The labels do not have to come from humans. Most modern DPO uses preferences labelled by a strong model, and synthetic datasets such as UltraFeedback were central to DPO’s adoption.
Direct Preference Optimization vs RLHF on Compute, Stability and Tuning
DPO is simpler and more stable; PPO-based RLHF needs more compute and engineering, but can reach a higher peak. The table summarises what each method demands.
| Dimension | Offline DPO | Iterative / online DPO | RLHF with PPO |
|---|---|---|---|
| Models involved | Policy plus a frozen reference whose log-ratios can be cached in one forward pass before training | Policy, reference, plus a reward model or LLM judge to label fresh samples | Four models in memory: policy, reference, reward model and critic |
| Sampling during training | None | Responses sampled from the current model each round | Fresh rollouts every step |
| Main hyperparameters | β and learning rate; the original paper used β = 0.1 and a learning rate of 1e-6 by default | As offline DPO, plus a labeller for self-generated responses | Separate actor and critic learning rates, global rollout batch size, KL penalty coefficient, reward-score clipping, GAE settings |
| Stability | Reported by the DPO authors as stable with little hyperparameter tuning | Depends on annotator quality and bias | Described as complex and often unstable |
| Typical ceiling | Solid model, lower peak, limited by dataset coverage | Surpasses offline DPO in several studies | Often the higher peak performance |
Xu et al. (2024) found three techniques mattered for PPO’s best results: advantage normalisation, large-batch training and an exponential moving average update of the reference model, with batch size giving the biggest benefit on code generation. On β the two methods agree: in the same study, too large a β hurt both, and 0.1 performed well across models and tasks.
Our reading of the table: for a small team with a clean preference set and no RL training framework, offline DPO is the sensible default, because the iterate-and-evaluate loop is a supervised training job they already know how to run.
When Is RLHF Still Worth the Extra Infrastructure?
RLHF with PPO generates responses beyond the preference dataset during training, while DPO is limited by that dataset’s coverage. The evidence is mixed, so it is worth reading both sides.
On the side of PPO, Xu et al. (2024) report PPO consistently beating DPO across their experiments, with state-of-the-art results on the hardest code-competition tasks. On CodeContest, DPO failed to generate any correct code after one epoch, and iterative DPO did not improve on the SFT model even with near-perfect test-case feedback. That is one study’s result, but the mechanism they propose is general: PPO can generate responses beyond the preference dataset and regularise them with the KL term, while DPO can raise the probability of out-of-distribution responses it never saw labelled. Ivison et al. (2024) found PPO ahead of DPO by up to 2.5% in maths and 1.2% in general domains.
Two further studies point to on-policy sampling. Tang et al. (2024) found offline algorithms train policies that are good at pairwise classification but worse at generation, while online algorithms show the opposite pattern. Tajwar et al. (2024) found methods that use on-policy sampling or push down the likelihood of specific responses generally outperform offline and maximum-likelihood objectives.
On the other side, the original DPO paper found DPO exceeded PPO-based RLHF on sentiment control and matched or improved it on summarisation and single-turn dialogue. On TL;DR summarisation, DPO reached a GPT-4 win rate of about 61% against PPO’s 57%, was more robust to sampling temperature, and its reward-versus-KL frontier dominated PPO’s.
Our judgement is that the pattern is about task shape rather than method quality. Where correctness can be checked automatically (code that compiles and passes tests, maths with a verifiable answer) and the model has to discover outputs the dataset never contained, on-policy RL pulls ahead. For style, tone and helpfulness on a well-covered prompt distribution, offline DPO is the simpler route, and in the original paper it matched or beat PPO on summarisation and single-turn dialogue.
Iterative DPO, IPO and DPOP Sit Between the Two
The choice is not binary. Iterative DPO samples responses from the current model, labels them with a reward model or annotator, and trains another round, which Xu et al. (2024) recommend for reducing distribution shift. Online AI feedback goes further, asking an LLM annotator to pick between two fresh samples at every iteration, and Guo et al. (2024) found it beat both offline direct alignment and RLHF in human evaluation.
Other variants change the loss itself. IPO, from Azar et al. (2023), starts from the observation that DPO still relies on substituting pointwise rewards for pairwise preferences, and builds an objective expressed in pairwise preferences directly. DPO-Positive (DPOP), from Pal et al. (2024), targets the failure where standard DPO lowers the likelihood of preferred examples as long as the relative probability still rises.
How Do I Evaluate Whether an Alignment Run Worked, Rather than Trusting a Lower Training Loss?
A falling DPO loss shows the model has widened the gap between chosen and rejected log-ratios relative to the reference. It does not guarantee the model ranks responses better. The DPO training loss is an optimisation diagnostic, not evidence of alignment. Several independent results explain why:
- The proxy loss does not guarantee ranking. DPO minimises a smooth surrogate, not ranking error, and Mohri and Zhong (2026) showed a model can drive the loss close to its floor by making the two scores nearly equal, without learning which response is better.
- Chosen likelihood can fall. Because only the margin matters, the model can lower the loss by pushing the rejected probability down faster than the chosen one while the chosen probability also drops, a pattern called likelihood displacement.
- Rankings stay poor. Chen et al. (2024) found most preference-tuned models they studied reached under 60% ranking accuracy on common preference datasets, and attributed the gap to a DPO objective ill-suited to fixing ranking errors already present in the reference model.
- Length grows first. Within the first 10% of an epoch, a standard DPO run in one 2024 study produced answers almost twice as long as the SFT model, peaked in win rate there, and then only grew in length and KL divergence.
A 2026 paper goes further, arguing that DPO’s equivalence to RLHF holds only when the reference policy already prefers the human-preferred response; when it does not, DPO optimises relative advantage over the reference and the failure is invisible in the loss curve. In its Llama-3-8B-Instruct analysis, nearly half of all pairs fell in that regime. The authors say their proposed fix still needs verification at larger scale, so we read the paper as an argument for monitoring rather than a settled measurement.
Neither Method Is Immune to Reward Hacking
Removing the reward model does not remove over-optimisation. In RLHF, the reward model is an imperfect proxy, and Gao, Schulman and Hilton (2022) measured how optimising it too hard degrades the true objective. Rafailov et al. (2024) found direct alignment algorithms show similar degradation at higher KL budgets, often before a single epoch is complete. Format is the other exploit: online iterative DPO and PPO amplified format biases from the reward model more than offline DPO, and fewer than 1% biased samples were enough to skew the reward model.
A Verification Checklist for DPO and RLHF Runs
- Hold out preference pairs and track ranking accuracy on them, not only on the training set; in Hugging Face TRL this is
rewards/accuracies, the share of pairs where the implicit reward for the chosen completion beats the rejected one. - Log chosen and rejected rewards separately, not just
rewards/margins; a margin that grows while the chosen reward falls is likelihood displacement. - Measure win rate on held-out prompts with a judge that played no part in training, such as a separate reward model or a GPT-4-class judge, as Xu et al. (2024) did.
- Control for length: compare win rates at matched response length, since length regularisation in one study gave close to 20% higher win rates on HH at matched lengths.
- Plot win rate against KL from the reference across checkpoints; higher reward at much higher KL is not necessarily an improvement.
- Spot-check with humans where the stakes justify it; in the original DPO paper, GPT-4 agreed with humans about as often as humans agreed with each other, which makes it a reasonable proxy for human evaluation.
- For online runs, watch the RLHF reward: TRL’s online DPO trainer logs
rlhf_reward, which should keep rising if training works.
Choosing checkpoints by held-out win rate rather than final loss follows from points 3 and 4, because DPO often reaches its best performance within a few hundred gradient steps.
Where Does Preference Alignment Sit Relative to Supervised Fine-Tuning, and Can It Be Skipped with Prompting or Retrieval Instead?
In the InstructGPT pipeline, preference alignment came after supervised fine-tuning. The InstructGPT pipeline fine-tuned GPT-3 on labeller demonstrations first, then used rankings of model outputs to fine-tune further with RLHF. DPO’s default reference policy is the model as it stands before DPO training, and the 2026 conditional-equivalence result found DPO degrades when that reference is misaligned.
Whether to align at all depends on what the gap is. If the model lacks facts or current documents, preference tuning will not supply them, and retrieval-augmented generation is the more direct route. If the model knows enough but answers in the wrong style, order or level of caution, and prompting alone does not hold that behaviour reliably, preference data is the signal that encodes “this answer over that one”. Whether a use case needs preference alignment at all, and whether offline DPO suffices, is one of the feasibility questions settled before a generative AI fine-tuning programme is committed to.
The question a team has to answer before picking DPO or RLHF is narrower than it looks: can our preference data cover the outputs we want the model to produce? If the model has to find answers nobody wrote down, someone will end up building the online loop, whether it is called RLHF, iterative DPO or online AI feedback.
FAQ
What Is a Reward Model in RLHF?
A reward model takes a prompt and response as text and outputs a single number, the reward. It is trained on human preference comparisons with a cross-entropy loss that works like logistic regression on which response was preferred.
What Is the Difference Between Exploration and Exploitation in Reinforcement Learning?
Exploitation means choosing actions already known to score well; exploration means trying new actions that might score better. Online RLHF explores by sampling fresh responses during training, which offline DPO cannot do; the basics are covered in our piece on how agents learn through trial and error.