Study notes · 2026-10-16 · Advanced

From reinforcement learning from human feedback to DPO and GRPO: how models are taught preferences

An LLM that has only been pretrained is good at 'guessing the next word' but has never learned to follow user intent, so post-training methods such as RLHF, DPO and GRPO teach it human preferences or correctness signals. This note covers RLHF, which uses a reward model and PPO; DPO, which removes the reinforcement learning stage; and GRPO, which compares responses within a group without a separate value model. It summarizes what each method changed and the issues that remain.

Contents

Why it was needed: making models bigger does not make them follow instructions
Core idea: ask people to pick between two dishes instead of cooking
How it works: the steps of the three methods
How it developed: what each of the three founding papers introduced
Where it is heading: agents, security and efficiency
Limitations and open issues
Further study

Key points

The pretraining objective (next-token prediction) and 'following user instructions helpfully and safely' are different goals, and RLHF bridges this gap with human preference comparisons.
RLHF consists of three stages: SFT → reward model training → PPO reinforcement learning. In InstructGPT, a 1.3B model was preferred over the 175B GPT-3.
DPO optimizes the same objective without separate reward model and reinforcement learning stages, using only a simple classification loss on preference pair data.
GRPO is a variant of PPO that groups multiple responses to the same question and compares them with each other, so it needs no separate value model (critic) and saves memory.
Recent research addresses step-level credit assignment in long agent tasks, security threats such as preference data poisoning, and training stability and memory efficiency.

Why it was needed: making models bigger does not make them follow instructions

LLMs (large language models) learn to 'guess the next token' by reading text from the internet. The InstructGPT paper paper points out that this objective differs from 'following user instructions helpfully and safely', and calls this state misaligned. As a result, models may make things up (hallucination), write biased or harmful text, or simply not follow instructions.

The same paper took as its starting point that scaling up models does not solve this problem on its own. Supervised fine-tuning (SFT), in which people write out the desired behavior as demonstrations, is one approach, but writing many examples of good responses is costly. The DPO paper paper also notes that relative judgments such as 'which of two responses is better' are easier to collect than demonstrations written by experts.

This is where RLHF (reinforcement learning from human feedback) came in. It turns the preferences people choose into a reward signal and post-trains the model to earn more of that reward. This line of work then continued into DPO and GRPO, around the questions of 'how to build the reward and how to optimize it efficiently'.

Core idea: ask people to pick between two dishes instead of cooking

Think of a cooking contest. Asking judges to cook a perfect dish themselves is hard, but asking them to taste two dishes and pick the better one is quick. RLHF collects this 'picking' at scale, builds a 'machine judge' (the reward model) that imitates the judges' criteria, and trains the cook (the language model) to score well with that judge.

DPO does not build a separate machine judge. As the subtitle of the DPO paper paper, 'Your Language Model is Secretly a Reward Model', suggests, the cook adjusts directly to 'serve good dishes more often and bad dishes less often'. Mathematically it targets the same objective as standard RLHF, but with far simpler steps.

GRPO has the cook make several dishes for one customer (question), compares the scores of those dishes against each other, and reinforces the ones better than average. The key point is that it uses comparison within the same group as the baseline, without a model that separately predicts 'how many points this question is worth to begin with'.

How it works: the steps of the three methods

RLHF works in the three stages laid out in InstructGPT paper. First, GPT-3 is trained with supervised learning on demonstrations of desirable responses written by labelers (SFT). Second, labelers rank several model outputs for the same prompt, and this data is used to train a reward model (RM) that predicts 'which output labelers would prefer'. Third, this reward model serves as the reward function, and the SFT model is fine-tuned again with a reinforcement learning algorithm called PPO. During this process, a constraint is also applied to keep the model from drifting too far from the original model.

DPO paper merges the second and third stages into one. The researchers showed that by expressing the reward model in a new way, the corresponding optimal policy (the way the model produces answers) can be obtained directly in closed form. So given only pairs of 'preferred and rejected responses', training comes down to a single binary classification loss that raises the relative probability of the preferred response and lowers that of the rejected one. There is no need to sample responses from the model during training. The paper explains that using a plain probability ratio alone causes the model to degenerate, and DPO prevents this by weighting each example differently.

GRPO (Group Relative Policy Optimization) is a PPO variant introduced in DeepSeekMath paper, aimed at improving mathematical reasoning while reducing PPO's memory use. According to descriptions in later studies, GRPO-style methods generate a group of multiple Rollouts (responses the model actually generates) for the same prompt and judge how good each response is using reward statistics within that group. For example, Fed-GRPO paper makes use of reward statistics that arise naturally during GRPO training, and MCGPO paper and CCPO paper, which deal with group-based reinforcement learning, describe this training as done without a separate value model (critic).

How it developed: what each of the three founding papers introduced

InstructGPT 2022 paper applied RLHF to broad instruction following and showed that it worked. The team hired 40 contract labelers to build the data, and in human evaluations, outputs of the 1.3B-parameter InstructGPT were preferred over those of the 175B GPT-3, a model more than 100 times larger. The 175B InstructGPT was preferred over the 175B GPT-3 85±3% of the time, and the rate of making up content not in the input fell from 41% for GPT-3 to 21%.

DPO 2023 paper targeted the problem that RLHF is 'a complex and often unstable procedure'. It removed the burden of training multiple language models and sampling repeatedly inside the training loop, and optimized the same objective without reinforcement learning. In experiments with models of up to 6B parameters, it beat PPO-based RLHF at controlling the sentiment of generated text, and matched or improved response quality in summarization and single-turn dialogue.

DeepSeekMath 2024 paper took a somewhat different direction. Its goal was mathematical reasoning rather than human preference: it continued pretraining a 7B model on 120B math-related tokens and then applied reinforcement learning with GRPO. This 7B model scored 51.7% on the competition-level MATH benchmark without external tools or voting. Where RLHF used 'human comparisons' as the reward, GRPO opened a path to making PPO-style methods lighter and using them for reasoning training. In later research, this line of work is mentioned alongside RLVR (reinforcement learning with verifiable rewards), which uses verifiable rewards such as whether an answer is correct.

Where it is heading: agents, security and efficiency

The first direction is credit assignment for agents doing long tasks. GRPO-style methods often receive only the final outcome (success or failure) as the reward, making it hard to tell which of dozens of steps were good actions. CCPO paper finds steps in other Trajectories (action records) where the same observation occurred and compares them, giving more weight the more similar the context is. It also considers 'how much progress the step actually made', so that redundant actions inside successful Trajectories are filtered out. With Qwen2.5-Instruct 1.5B and 7B models, this method raised the success rate to as high as 97.4% on ALFWorld and 79.4% on WebShop, up to 21.6 and 18.5 points higher than GRPO, respectively.

The second direction is the security of preference data. As DPO became widely used, the preference data itself became a target for attacks. DPOison-VL paper presented an attack on VLMs (vision-language models) trained with DPO that leaves prompts, responses and preference labels unchanged and adds imperceptible perturbations to only some training images. Poisoning just 2% of the training data brought the attack success rate as high as 98%. It is a warning that data poisoning that slips past human review can quietly change alignment outcomes.

The third direction is denser signals and efficiency. FSPO paper argues that a single outcome score in RLVR only teaches the whole response coarsely, so it stores past next-token distributions as fixed targets to provide denser, token-level training signals. It does not keep a separate Teacher model running. Within a 5-hour training budget, it scored about 4 percentage points higher on average than the strongest baseline, and on Olmo3-7B-Instruct it reduced peak memory from about 70GB to 56GB.

Notable figures from recent research

Up to 21.6 pointsCCPO, vs GRPO on ALFWorld

2% → 98%DPOison-VL, poisoning rate and attack success rate

70GB → 56GBFSPO, peak memory reduction

These are figures each study reported in its own experimental setup and do not represent real-world service performance.

Limitations and open issues

The most fundamental issue is 'whose preferences?' InstructGPT paper itself states that this method aligns the model not to 'human values' in a broad sense but to the stated preferences of a specific group (mainly labelers and researchers). In the same paper, harmful outputs decreased, but there was no clear improvement on the bias measures Winogender and CrowS-Pairs, and performance on public NLP datasets such as SQuAD and DROP dropped during RLHF. The researchers reported that a variant mixing in pretraining data (PPO-ptx) reduced this drop.

The second is reward quality. RLHF maximizes a learned reward model, so if the reward model imitates people's true judgments poorly, the model can exploit the gaps (reward hacking). The constraint that keeps the model from drifting too far from the original model is one safeguard against this. DPO has no separate reward model, but it depends directly on the preference data. As DPOison-VL paper shows, if that data is poisoned, the direction of alignment changes.

The third is the coarseness of outcome rewards. Methods like GRPO that look only at whether the final answer is correct cannot say which steps contributed in a long task. This is why CCPO paper and FSPO paper were developed. Also, most experimental figures come from specific benchmarks and specific model sizes, so they should not be taken as real-world service performance. Whatever method is used, evaluating and taking responsibility for the final behavior remains with people.

Further study

Start with Figure 2 (the three stages of SFT, reward model and PPO) and the Section 1 introduction of InstructGPT paper to get the overall shape of RLHF. Comparing how truthfulness, toxicity and bias each changed in the results summary shows 'what alignment changes and what it cannot'.

Next, read Figure 1 and the introduction of DPO paper. The figure places the RLHF pipeline and the DPO pipeline side by side, so you can see at a glance what was removed. If the equations feel heavy, it is fine to take away just the intuition of 'raising the relative probability of the preferred response'.

Then check the context in which GRPO appeared in DeepSeekMath paper (mathematical reasoning, reducing PPO memory). For recent work, we recommend going on to CCPO paper, which addresses GRPO's limits in agent training, and DPOison-VL paper, which deals with preference data security.

References

Foundational papers

Recent research

Written by AI from the listed references and reviewed. Please check the original sources for details.