AI Trend Research

← AI glossary

AI glossary · Advanced · Reinforcement learning · Language models

DPO

DPO (Direct Preference Optimization) is a method that trains an LLM directly on preference data, in which people chose the better of two answers, without a separate reward model or reinforcement learning stage.

About this term
Level
Advanced · Specialist terms that need the inner workings
Added
2026-10-08
In plain words It is like teaching a chef by directly showing them records of which of two dishes customers chose and saying "make it like this one," instead of training a separate judge to score the chef.

How it works

First, data is collected containing two answers to the same question and a record of which one people preferred. DPO trains the model directly to raise the probability of producing the preferred answer and lower the probability of producing the non-preferred one. The original model before training is used as a reference to keep the model from drifting too far from its original behavior. The process resembles ordinary supervised learning, so implementation and operation are simple.

How it differs

Conventional RLHF goes through two stages: first training a reward model on preference data, then running reinforcement learning using that reward model's scores. DPO merges these two stages into a single training step, skipping the reward model and reinforcement learning. As a result, it needs fewer computing resources and suffers less from the instability typical of reinforcement learning.

RLHF vs. DPOPreference data → Train reward model; Train reward model → Reinforcement learning(Provide scores); Reinforcement learning → Tuned LLM; Preference data → Direct training; Direct training → Tuned LLMRLHFDPOPreference dataA is better than BTrain reward modelReinforcementlearningTuned LLMPreference dataA is better than BDirect trainingPreferred answer probability↑ Non-preferred answerprobability↓Tuned LLMProvide scores
RLHF vs. DPO The top shows RLHF, which goes through a reward model and reinforcement learning; the bottom shows DPO, which trains the model directly on preference data.

Things to know

DPO is an off-policy method that learns only from preference data collected in advance, so results depend heavily on data quality and coverage. Some point out that it may improve performance less than reinforcement learning methods in which the model improves by having answers it newly generates during training evaluated. So in practice, it is combined to suit the situation, such as using DPO after SFT or together with other reinforcement learning techniques.

Example

Organizations releasing open-weight LLMs often use DPO when making "chat" versions that follow instructions well and respond in a style people prefer. This is because labs and companies that cannot afford to train a separate reward model and run reinforcement learning can still refine a model in the direction they want with relatively few resources, as long as they have preference data. Fine-tuning tools also often provide DPO as a standard feature alongside SFT.

Related terms

RLHF Reinforcement learning Post-training SFT Alignment Fine-tuning