AI Trend Research

← AI glossary

AI glossary · Advanced · Reinforcement learning · Safety & alignment

RLHF

RLHF (reinforcement learning from human feedback) is a method that collects people's judgments about which of an AI's several answers is better, and uses reinforcement learning to refine the model so it answers in the ways people prefer.

About this term
Level
Advanced · Specialist terms that need the inner workings
Added
2026-10-08
In plain words It is similar to a chef revising recipes based on customer rating cards. Since customers cannot taste every dish themselves, the chef sets up a separate 'taster' who imitates the customers' palate and practices to that taster's scores. The limitation is that the taster cannot perfectly reproduce the customers' tastes.

How it works

First, a model that has finished pretraining is shown model answers written by people and learns basic conversational behavior through SFT (supervised fine-tuning). Next, the model produces several answers to the same question, and people rank which ones are better. This comparison data is used to train a separate reward model that scores 'which answer a person would prefer'. Finally, reinforcement learning is repeated: the model generates answers, the reward model scores them, and the model is adjusted in the direction that raises the score.

The three stages of RLHFPretrained model → SFT; SFT → People compare answers(Answers); People compare answers → Reward model(Training); SFT → Generate answers(Starting point); Generate answers → Reward score; Reward model → Reward score(Scoring); Reward score → Model update; Model update → Generate answers(Repeat)① Preparation · Building the reward model② Reinforcement learningPretrained modelSFTTrained on modelanswersPeople compareanswersRank which answer isbetterReward modelGenerate answersReward scoreModel updateToward higher scoresAnswersTrainingStarting pointScoringRepeat
The three stages of RLHF The top row is the preparation stage, where people compare the SFT model's answers to build a reward model. The bottom is the reinforcement learning stage, which repeats answer generation, reward scoring and model updates.

Why it matters

A language model with only pretraining just continues text plausibly; it does not really know how to answer questions helpfully or refuse dangerous requests. RLHF is a way to bring human judgments of what a 'good answer' is into the model, and it is regarded as a key technique that made conversational AI usable. Conversational services such as ChatGPT are widely known to have used this approach. It is also an important starting point for alignment research, which aims to make AI fit human intentions and values.

Things to know

It requires collecting many human evaluations, which takes considerable cost and time. Because the reward model only imitates human judgment, reward hacking can occur, where the model exploits loopholes in the reward model to raise only its score. Some also point out that if evaluators prefer confident, pleasant-sounding answers, the model develops a sycophantic tendency to agree with whatever the user says. For these reasons, alternatives are also used, such as DPO, which trains directly on preference data without a reward model, and RLVR, which gives rewards on problems whose answers can be checked automatically.

Example

In conversational AI services, the 'like/dislike' buttons next to answers, or screens asking you to pick the better of two answers, resemble the way this kind of human preference data is collected. Model developers build reward models from answer comparisons made by professional evaluators, and use them during post-training to refine the model so it gives more helpful and less harmful answers.

Related terms

Reinforcement learning Post-training SFT DPO Alignment Reward hacking