Definition
Direct Preference Optimization, commonly shortened to DPO, is a method for adapting a language model from preference pairs. Each training example contains the same input with a preferred response and a rejected response. The objective favors the chosen response over the rejected response relative to a fixed reference policy.
DPO optimizes this objective directly. It avoids the separate learned reward model and reinforcement-learning optimization loop used in a common form of RLHF.
Simple example
Reviewers compare two answers to a support question and mark the answer that follows policy while clearly stating uncertainty. Across many such pairs, DPO training updates the model so preferred responses become more likely relative to rejected ones.
Why it matters
Preference data captures relative judgments that may be easier to provide than one ideal target response. DPO offers a comparatively direct training pipeline for using those judgments and can reduce some infrastructure associated with reward-model training and online reinforcement learning.
DPO still requires careful data collection, reference-model choices, optimization, and evaluation. Preference labels can be inconsistent or reflect position effects and annotator bias.
One important nuance
DPO differs from the reward-model-plus-reinforcement-learning pipeline commonly associated with RLHF. Calling DPO simply “RLHF without reinforcement learning” obscures differences in their objectives, data flow, and training behavior. DPO does not eliminate problems caused by inconsistent or incomplete preference data. A model trained with DPO can still overfit superficial patterns, so evaluate it on held-out tasks and check for regressions.