“We did RLHF” is not an alignment strategy. It is a sentence that hides an architecture decision — one that was probably made by default, because a reward model plus PPO is the pipeline everyone read about first.
Alignment tuning is a choice of optimisation target, not a fixed step you tick off after supervised fine-tuning. Reinforcement Learning from Human Feedback (RLHF) trains a separate reward model on human preference comparisons, then optimises the language model’s policy against that reward with reinforcement learning. Direct Preference Optimization (DPO) reparameterises the same preference objective so that the model is trained directly on pairwise preference data with a supervised-style loss. Same intent, different machinery — and the machinery is where budgets, timelines, and failure modes live.
The divergence point is the reward model. Once you remove it, you also remove reward-model drift, reward hacking against a proxy, and most of the RL infrastructure and tuning surface. What you give up is the online exploration RLHF permits: the ability to keep generating fresh samples, score them, and improve against a live signal.
What the DPO loss actually optimises
Both methods start from the same assumption about human preferences: given a prompt and two candidate responses, a human picks one. RLHF fits a reward model to those choices — typically a Bradley–Terry style objective over the pairs — and then treats that model as a stand-in for the human, optimising the policy against it while a KL penalty keeps the policy from wandering too far from the supervised-fine-tuned reference.
DPO’s insight is that this two-stage construction has a closed-form shortcut. If the optimal policy under a KL-constrained reward objective can be written as a reweighting of the reference policy, then the reward can be written in terms of the policy and the reference policy instead. Substitute that back into the preference likelihood and the reward model disappears from the expression. What remains is a loss over pairs that compares the log-probability ratio the tuned model assigns to the chosen response against the ratio it assigns to the rejected one, both measured relative to a frozen reference copy of the same model.
The practical consequences follow directly from the maths:
- You need two forward passes per pair (policy and frozen reference), no sampling loop, no critic, no value head.
- The reference model is the KL anchor. It is not optional, and drifting far from it is the main way DPO runs degrade.
- The temperature-like coefficient (usually written β) controls how hard the loss is allowed to push away from the reference. It is the single most consequential hyperparameter in the method.
This is why DPO feels like supervised fine-tuning to implement. In PyTorch, a DPO step is a batch of four sequences (chosen and rejected, under policy and reference), a log-prob difference, and a logistic loss. Libraries such as Hugging Face TRL expose it as a trainer class that slots into an existing fine-tuning script. An RLHF loop, by contrast, needs generation, reward scoring, advantage estimation, and a PPO update — four components that each have their own way of going quietly wrong.
Why “we did RLHF” is the wrong unit of decision
The failure class we see most often is not a bad alignment run. It is an alignment programme committed to before anyone counted the preference pairs or the GPU-hours.
A team decides the model needs to “behave better”, reaches for the pipeline they read about, and discovers three weeks in that they have four thousand noisy preference pairs, no RL tooling, no one who has tuned PPO before, and a reward model that scores fluent nonsense higher than terse correct answers. That last symptom — reward hacking against a proxy — is not a bug in their implementation. It is the structural risk of optimising against a learned reward model at all. The policy is doing exactly what you asked: maximising a proxy. The proxy is the problem.
Removing the reward model removes that failure mode. It does not remove all failure modes, and it introduces one of its own: because DPO only ever sees the pairs you collected, it cannot discover that the model has found a new bad behaviour outside the distribution of your dataset. RLHF’s rollout loop can, at least in principle, because it keeps sampling from the current policy.
So the honest framing is a trade: offline stability and a short iterate-evaluate cycle, against online exploration and the operational cost that comes with it.
Decision matrix: DPO vs RLHF
Evidence class for this table: observed-pattern — drawn from alignment work across our generative-AI engagements, not a published benchmark. Portability is limited; treat the thresholds as planning heuristics, not thresholds you can cite in a design review.
| Dimension | DPO | RLHF (reward model + PPO) |
|---|---|---|
| Training stages | One (plus the SFT you already have) | Two (reward model, then policy) |
| Data required | Pairwise preferences, offline, fixed | Pairwise preferences plus capacity to score fresh rollouts |
| Typical pair volume before behaviour shifts measurably | Low thousands of clean pairs | Similar for the reward model, but the policy stage keeps consuming signal |
| Hyperparameters that materially matter | β, learning rate, reference checkpoint choice | Reward-model config, KL coefficient, clip range, rollout length, batch composition, value-loss weight |
| Compute profile | Two forward passes per pair; no generation | Generation-dominated; rollouts are the bottleneck |
| Iterate-evaluate cycle | Hours on comparable hardware | Days, in most setups we’ve worked with |
| Dominant failure mode | Over-optimisation away from the reference; dataset blind spots | Reward hacking; reward-model drift; PPO instability |
| Can discover new bad behaviour during training | No | Yes, in principle |
| Realistic team size to run it | Small, no RL specialist | RL experience effectively required |
The row that decides most cases is the second-to-last one. If you genuinely need iterative on-policy improvement against a live reward signal — because behaviour must be shaped in regions of the output space your static dataset does not cover — the reward model and the RL loop are the price of admission. If you do not, you are paying for exploration you never use.
What preference data do you actually need?
Format first, volume second. A DPO dataset is a list of records with a prompt, a chosen response, and a rejected response. That is the whole schema. The subtleties are in how the pairs were produced.
Three properties matter more than raw count:
Label consistency. Pairs where two annotators would disagree contribute gradient in both directions and cancel out. In practice, a smaller set with a tight annotation rubric outperforms a larger set collected with a vague instruction like “pick the better answer”. This is the single highest-leverage thing a small team controls.
On-policy proximity. Rejected responses should look like things your model actually produces. If the rejected side was generated by a much weaker model, the loss learns to distinguish model families rather than quality, and the tuned model gets very good at not sounding like the weak model — which is not what you asked for.
Coverage of the behaviours you care about. DPO cannot align what it never sees. If refusal behaviour, citation formatting, and tone are three distinct goals, the dataset needs pairs that isolate each of them. Mixing them into one undifferentiated pile produces a model that moves on all three axes unpredictably.
Our working rule: get the rubric right, generate rejected responses from the model you are actually tuning, and start measuring at the low-thousands scale rather than waiting for a large dataset. Refreshing a clean dataset and rerunning is cheap under DPO; that is the point of removing the RL loop.
How do you tell whether an alignment run worked?
Not from the training loss. A DPO loss that goes down means the model assigns relatively more probability to chosen responses than rejected ones on your training pairs. It says nothing about whether the model is better.
The measurement that matters is win-rate on a held-out preference set the tuning run never touched. Build it before you train, keep it frozen, and score it with a consistent judge — human annotators for the questions you care most about, a strong model-as-judge for the volume tier, with a periodic human audit of the judge’s calls. Report the win-rate against the pre-tuning checkpoint, not against an abstract ideal.
Two guardrail metrics belong next to it:
- KL divergence from the reference model. Rising KL alongside a rising win-rate is normal; rising KL with a flat win-rate means you are burning capability for nothing.
- A capability regression check. Run whatever task suite reflects your actual product — extraction accuracy, instruction-following, code compilation rate. Alignment tuning that improves preference win-rate while degrading the underlying task is a net loss, and it is easy to miss if win-rate is the only number on the dashboard.
The ROI signature of a well-run DPO pipeline is fewer tuning runs to reach a target win-rate, and a pipeline a small team can rerun end-to-end when the preference dataset is refreshed. That reproducibility is worth more over a year than any single run’s numbers.
Where preference alignment sits relative to fine-tuning
Preference alignment is not a substitute for supervised fine-tuning, and it is often not the first thing to reach for at all.
The sequence that holds up: get the base capability from supervised fine-tuning on demonstrations of the task done correctly. Then, if the remaining gap is about which of several acceptable outputs the model prefers — tone, verbosity, refusal boundaries, formatting discipline — preference alignment is the right tool. If the model simply cannot do the task, no amount of preference data will teach it; that is an SFT or a retrieval problem.
And a meaningful share of behaviour problems dissolve under prompting or retrieval. If the model gives the wrong answer because it lacks the document, a retrieval pipeline fixes it and an alignment run does not. If the output format is wrong, a constrained-decoding or structured-extraction approach is cheaper and more reliable than teaching format preference through pairs — the same reasoning behind choosing a purpose-built structured-extraction library over a tuned model. Alignment tuning earns its cost when the behaviour you want is genuinely a preference over acceptable outputs, and when you can express that preference in pairs more easily than in instructions.
That ordering question — does this use case need preference alignment at all — is one of the things we work through in our generative AI engineering practice before a client commits budget to a fine-tuning programme. Getting the answer wrong is expensive in a specific way: you build a tuning pipeline, it works, and it solves a problem you did not have.
FAQ
What is Direct Preference Optimization (DPO), and how does it differ from RLHF?
DPO trains a model directly on pairwise preference data using a supervised-style loss, comparing the log-probability ratios the model assigns to chosen versus rejected responses relative to a frozen reference copy. RLHF instead fits a separate reward model to the same preference data and then optimises the policy against that reward with reinforcement learning. Both target the same preference objective; DPO reparameterises it so the reward model is no longer needed.
What does the DPO loss actually optimise, and why does it not need a separate reward model?
Under a KL-constrained reward objective, the optimal policy can be written as a reweighting of the reference policy — which means the reward can be expressed in terms of the policy and reference instead. Substituting that back into the preference likelihood eliminates the reward model from the expression. What remains is a logistic loss over pairs requiring two forward passes each, no sampling loop, no critic, and no value head.
What preference data do I need for DPO — how many pairs, in what format, and how clean must the labels be?
The format is minimal: prompt, chosen response, rejected response. Label consistency matters more than volume — a smaller set collected under a tight annotation rubric typically outperforms a larger set gathered under a vague instruction. Rejected responses should come from the model you are actually tuning, and the dataset must cover each behaviour you want to change separately. Starting to measure at the low-thousands scale is a reasonable planning heuristic, not a benchmarked threshold.
When is RLHF still the better choice despite the extra reward-model and RL infrastructure?
When you need iterative on-policy improvement against a live reward signal — specifically, when the behaviours you must shape live in regions of the output space a static preference dataset does not cover. RLHF’s rollout loop keeps sampling from the current policy, so it can encounter and correct newly emerging failure modes; DPO cannot align what it never sees. That capability requires RL experience on the team and a compute profile dominated by generation.
How do DPO and RLHF compare on compute cost, training stability, and number of hyperparameters to tune?
DPO removes an entire training stage and the rollout loop, collapsing the hyperparameters that materially matter to roughly β, learning rate, and reference-checkpoint choice. RLHF adds reward-model configuration, KL coefficient, clip range, rollout length, and value-loss weighting. In the setups we’ve worked with, that difference shortens the iterate-evaluate cycle from days to hours on comparable hardware — an observed pattern, not a published benchmark.
How do I evaluate whether an alignment run worked, rather than trusting a lower training loss?
Use win-rate on a held-out preference set the tuning run never touched, scored against the pre-tuning checkpoint with a consistent judge. Pair it with two guardrails: KL divergence from the reference model, and a capability regression check on the task suite that reflects your actual product. A falling training loss only says the model separates your training pairs better; it says nothing about whether the model improved.
Where does preference alignment sit relative to supervised fine-tuning, and can it be skipped with prompting or retrieval instead?
Supervised fine-tuning establishes the capability; preference alignment shapes which of several acceptable outputs the model favours. If the model cannot do the task at all, preference data will not teach it. And a meaningful share of behaviour complaints are really retrieval gaps or output-format problems, both of which are cheaper to solve with a retrieval pipeline or constrained decoding than with an alignment run.
The question to settle before the first tuning run
The interesting question is not “DPO or RLHF”. It is whether the behaviour you want to change is expressible as a preference over outputs the model can already produce. If it is, DPO is usually the shorter path and the one a small team can maintain. If it is not — if the model must be pushed into territory your dataset does not describe — then you are buying exploration, and the reward model and RL loop are what exploration costs.
Answer that first, and the method selection stops being an architecture decision made by default.
