RLVR: Building LLM Reasoning with Verifiable Rewards

Why it was needed: the limits of imitation and human grading

The most familiar way to improve LLM reasoning is SFT (supervised fine-tuning): showing well-written solutions and having the model copy them. However, according to Reasoning-Table paper, a study on table reasoning, SFT produced results, but because of the biases that come with imitation learning, its generalization and robustness on unseen problems were weak. This means imitating demonstrated solutions and solving problems on one's own are different abilities.

There is also RLHF (reinforcement learning from human feedback), which uses human judgments as rewards. But it requires building a separate reward model trained on human preferences, and if this reward model is wrong, the model can exploit its loopholes, leading to reward hacking. For problems with clear right and wrong answers, like math proofs or code, there is no need for a grader that imitates human taste.

This led to the idea of RLVR (Reinforcement Learning with Verifiable Rewards). For math problems, the final answer is compared with the correct answer; for code, the tests are run. According to an RLVR study paper, interest in RLVR surged once the GRPO (Group Relative Policy Optimization) algorithm used by DeepSeek-R1 made long CoT (Chain-of-Thought, step-by-step reasoning) possible.

Core idea: self-study with an answer key but no worked solutions

RLVR is like a student working through a problem book with only an answer key and no worked solutions. No one teaches the solution process. Instead, the student solves the same problem several times, checks against the answer key, and next time uses the approach that worked more often. With repetition, habits that lead to correct answers gradually take hold.

The key point is that the reward comes from a program, not a human judgment. The graders are rules that can be checked mechanically: matching the correct answer, passing unit tests, matching SQL execution results, or following a required output format. Reasoning-Table paper also reported outperforming SFT on several benchmarks using only 'simple rule-based outcome rewards.'

This raises a question. If only the final answer is graded, won't a solution that reached the right answer by chance through a flawed process also be rewarded? An RLVR study paper built a theoretical framework explaining why correct reasoning is encouraged even when the reward depends only on whether the answer is correct. The argument is that flawed processes reach the correct answer less often on average, so over many repetitions, correct processes are reinforced more.

How it works

A single training step proceeds roughly as follows. First, pick a problem that comes with an answer or tests. Second, the current model (the policy) generates several answers to the same problem. Each of these answers is called a rollout. Third, a verifier grades each rollout and gives a reward such as 1 if correct and 0 if wrong. Fourth, GRPO treats the answers to the same problem as one group and updates the model so that answers better than the group average appear more often and worse ones less often. This process repeats.

This approach is on-policy, meaning it learns from answers the current model generated itself. As the exploration study ASE paper points out, the model receives a direct learning signal only from trajectories (solution paths) that actually appeared in its own rollouts. So if all answers in a group are correct, or all are wrong, there is no difference to compare and almost nothing to learn. This is why problem difficulty matters.

The relative budget theory paper explains this with a single value. The relative budget ξ is the generation token budget H divided by the expected number of tokens the base model uses to reach its first correct answer. When ξ is close to 0, useful trajectories are rare and the number of samples needed explodes. In the balanced range around ξ≈1, training is most efficient. When ξ is very large, training is stable but the gain per iteration shrinks. In experiments, training efficiency was highest when ξ was 1.5–2.0, and reasoning performance also peaked there.

What various algorithms actually optimize has also been clarified. Work by Davis and Recht paper showed that many algorithms with binary rewards (1 if correct, 0 otherwise) are stochastic gradient ascent on a monotone transformation of 'the probability of producing the correct answer.' For rejection sampling the transformation is the logarithm, and for GRPO it is the arcsine of the square root. This means algorithms that look different are pursuing the same goal, raising the probability of a correct answer, with different weightings.

How the field developed: what has been found

The first question was whether RLVR really expands reasoning ability or merely makes the model get problems it could already solve right more often. A 2025 RLVR study paper re-examined Pass@K experiments (a metric that counts success if any of K attempts is correct). It added a new metric, CoT-Pass@K, which counts success only if both the final answer and the intermediate reasoning steps are correct. As a result, it presented evidence that RLVR can expand the boundary of reasoning on math and coding tasks and encourages correct reasoning from early in training.

The second line of work broadened the application areas. Also in 2025, Reasoning-Table paper applied reinforcement learning for the first time to table reasoning, covering table question answering, fact verification and Text-to-SQL. It scored 4.0% higher than Claude-3.7-Sonnet on table reasoning benchmarks, and a 7B model reached 68.3% on the BIRD dev dataset. It is an example showing that the same framework works outside math as long as answers can be checked by machine.

The third line is theoretical understanding. In 2025, work by Davis and Recht paper clarified what objective each algorithm optimizes. In 2026, the relative budget theory paper used a single value, ξ, to explain why RLVR's effectiveness varies with the task and compute budget. The question has shifted from 'it works when we try it' to 'when and why does it work.'

Side effects: overconfidence and reduced diversity

RLVR has two well-known side effects. The first is worse calibration. A well-calibrated model that is 70% confident is actually right about 70% of the time. According to the 2026 DCPO study paper, models trained with RLVR become overconfident even in wrong answers. Earlier work simply added a calibration objective to the training objective, but this study showed theoretically that there is a fundamental gradient conflict between the direction that raises accuracy and the direction that reduces calibration error. It therefore proposed DCPO, which fully decouples the reasoning and calibration objectives, and reported the best calibration while keeping accuracy similar to GRPO.

The second is reduced solution diversity. The TAR study paper showed that learning from only a limited number of on-policy rollouts gradually suppresses solution approaches that are semantically different but equally correct. Once a solution approach has become sufficiently unlikely to be sampled, standard RLVR almost never revives it. TAR stores diverse correct solutions produced by the base model in a small memory bank and replays them selectively, only when they are being suppressed. On 6 math reasoning benchmarks and Qwen2.5 models of two sizes, it achieved a better balance of performance and diversity than existing methods.

Flat-RLVR paper describes the same phenomenon as 'distribution sharpening.' As the model's output distribution concentrates too heavily on one point, rare but correct trajectories disappear. This study combined SAM (sharpness-aware minimization), which seeks points where the reward landscape is flat, with GRPO-family methods. As a result, it increased sample diversity while maintaining single-attempt accuracy. Both studies show that there is a tension between raising Pass@1 and preserving Pass@K.

Where it is heading now

The first direction is broadening exploration. ASE paper starts from the problem that repeatedly sampling from the same policy tends to concentrate on similar solution paths. It clusters the differences in internal activations between correct and incorrect answers to the same problem to build a set of 'exploration vectors', then applies a different vector to each Rollout so the model tries different solution paths. On Qwen3-1.7B-Base and Qwen3-8B-Base trained on DAPO-Math-17k, average Mean@8 across six math benchmarks was 4.87 and 1.87 percentage points higher, respectively, than GRPO with the same Rollout budget.

The second direction goes beyond math problems to long agent tasks. For such tasks, it is hard for people to write every task and its verification method by hand. In Explorer-Zero paper, a task-setter model explores the environment directly to create tasks, and records intermediate and final states as verification targets. This allows partial rewards even when a task is only partly completed. Without any human-annotated training tasks, a 30B agent reached a 59.67% task success rate on AppWorld, close to the result of reinforcement learning on human-annotated tasks (62.65%).

The third direction is Off-policy learning, that is, learning from answers produced by an earlier model rather than the current one. In real training systems, the side that generates answers and the side that trains easily fall out of sync. Correcting for this with importance weights increases variance on long responses. GGPO paper proposed a loss function that compares responses to the same prompt without these weights. It outperformed GRPO and AGRO under training-inference mismatch and on fixed offline data. Periodically updating the reference policy also mattered: without updates, the weighted average across five benchmarks was 32.96; with the default update schedule, it was 46.58.

Limitations and open questions

The most fundamental limitation is, as the name says, that it only works in 'verifiable' domains. It suits tasks where answers can be checked by machine, such as math, code, tables and SQL. But it is hard to apply as-is to tasks without a single correct answer, such as how persuasive a piece of writing is or how appropriate a piece of advice is. It is also always necessary to consider that a weak verifier lets the model exploit gaps in grading, for example by getting the format right while the content is wrong.

The debate over whether RLVR expands capabilities or just draws out existing capabilities more often is not fully settled. The paper study offered evidence that the reasoning boundary expands. On the other hand, TAR paper and Flat-RLVR paper showed that while single-attempt accuracy rises, rare but correct solution approaches can disappear. Conclusions can differ depending on which metric (Pass@1, Pass@K, CoT-Pass@K) is used.

From a practical standpoint, two points are worth remembering. One is the overconfidence problem shown by the DCPO study paper: even when a model trained with RLVR appears confident, that confidence is hard to take at face value. The other is the difficulty problem shown by relative budget theory paper: if you collect only problems that are too hard for the model's level and token budget, almost no learning signal comes out.

Further reading

If you are new to the topic, start with the RLVR study paper. It covers in one place the debate over what RLVR changes, the difference between Pass@K and CoT-Pass@K, and why grading only the answer still improves the process. Next, the short study by Davis and Recht paper shows that the algorithms ultimately pursue the same objective, and relative budget theory paper gives a sense of when training works well.

After that, reading DCPO paper, TAR paper and Flat-RLVR paper together on side effects shows what you miss by looking only at accuracy. If you want to know how to apply it beyond math, see Reasoning-Table paper on table reasoning.

For recent directions, pick by problem: ASE paper for exploration, Explorer-Zero paper for reward design in agent tasks, and GGPO paper for Off-policy issues that arise in training systems.