Generator–Verifier: a model that makes answers and a model that checks them

Why separate the model that makes answers from the model that checks them

This is part 1 of the Generator–Verifier series. It covers the roots of the idea of using a separate generator and verifier, and the line of research that followed. A generator is a model that produces answers. A verifier is a model or program that scores or judges whether an answer is correct.

The starting point is the observation that LLMs easily fall apart on multi-step reasoning because of a single mistake. The verifier study that introduced GSM8K paper pointed out that autoregressive models (models that generate the next token one at a time based on the previous tokens) have no mechanism for correcting their own errors during generation. So once a solution goes off track, it is hard to recover. The same study cited the trend that improving performance on hard problems through generation alone would require an excessive number of parameters, and argued for finding methods that scale more favorably.

The key idea that came out of this is the "generation–verification gap." Grading someone else's answer is usually easier than solving a math problem from scratch. The same study paper also described verification as generally a simpler task than generation. It added that there is a further advantage in having "options": being able to pick one of several candidates. As an analogy, the generator is an exam taker who writes several answer sheets, and the verifier is a grader who picks the most reliable one. Even if the exam taker's ability stays the same, a good grader raises the quality of the final submitted answer.

How much of this gap can actually be exploited, and what happens when the grader is wrong, are the questions that run through this whole series.

Thread ①: Best-of-N and outcome verifiers (ORM)

The simplest use is Best-of-N. For a given problem, the generator samples N solutions, and the one with the highest verifier score is chosen. The GSM8K verifier study paper released 8.5K grade-school math word problems. It then trained a verifier to judge whether model-generated solutions were correct. At test time, a fixed number of candidates was sampled and the solution ranked first by the verifier was submitted.

The results come down to two points. First, compared with a fine-tuning-only baseline, using a verifier improved performance about as much as scaling the model up roughly 30 times. Second, the verifier approach scaled much better as data increased. The study also reported that dropout (a technique that randomly turns off some connections during training) has a strong regularizing effect for both fine-tuning and verification.

A verifier like this, which scores a whole solution based only on whether the final answer is correct, is called an outcome-supervised reward model (ORM). As summarized in Let's Verify Step by Step paper, an ORM is trained only on the final result of a CoT (Chain-of-Thought, step-by-step solution). Its advantage is that labels are easy to create. For problems with a fixed correct answer, only the final answer needs to be compared, so training data can be produced without humans.

Its weakness comes from the same place. Let's Verify Step by Step paper pointed out that models supervised only on outcomes often reach correct answers through incorrect reasoning. An ORM cannot tell apart "a solution with the right answer" from "a solution whose reasoning is also correct."

Thread ②: Process rewards (PRM) and automatic step labels

A process-supervised reward model (PRM) judges separately whether each step of a solution is correct. Let's Verify Step by Step paper gave three advantages of PRMs. Feedback is more precise because it shows exactly where an error occurred. It is easier for humans to interpret. It directly rewards reasoning processes that humans endorse.

The study compared ORMs and PRMs using GPT-4-based models, and on the difficult MATH dataset the PRM clearly outperformed the ORM. The best PRM solved 78.2% of a representative subset of the MATH test set. The evaluation also took the verifier's perspective: the generator was held fixed, and performance was measured by how well the reward model picked correct solutions in Best-of-N. The study released PRM800K, a collection of 800,000 step-level human labels. It also used active learning (a method that picks the cases the model finds confusing and gets labels for them) to improve the data efficiency of process supervision by 2.6 times.

The problem is cost. Final answers can be graded automatically, but there is no simple way to automate step-by-step judgments. So the study relied on human labelers. Math-Shepherd paper targeted this bottleneck by training a PRM on automatically generated step-level supervision data, without human annotation. The resulting PRM was used in two places: as a verifier that reranks multiple outputs, and as the reward for step-by-step PPO (a reinforcement learning algorithm).

According to Math-Shepherd paper, step-by-step PPO raised Mistral-7B's accuracy from 77.9% to 84.1% on GSM8K and from 28.6% to 33.0% on MATH. Adding Math-Shepherd verification on top brought these to 89.1% and 43.5%, respectively. The main point of this thread is that the same PRM can serve both as a training signal and as a tool for picking answers at inference time.

Thread ③: Generative verifiers and LLM-as-a-Judge

The verifiers covered so far were mostly discriminative classifiers that take a solution and output a single score. The Generative Verifiers study paper argued that this approach does not use the text generation abilities of pretrained LLMs. It therefore proposed GenRM, a generative verifier that is trained on verification itself as next-token prediction. GenRM generates its answer to the question "Is this solution correct?" as text, and is trained jointly on solution generation and verification.

This lets the verifier keep the strengths of LLMs. It combines naturally with instruction tuning, and it can write out a CoT rationale before giving a judgment. By generating the same verification several times and deciding by majority vote, test-time compute (extra computation spent at inference time) can also be applied to the verification step. The Generative Verifiers study paper reported that GenRM outperforms discriminative verifiers and DPO verifiers, and also LLM-as-a-Judge (having an LLM do the grading without separate training). Best-of-N accuracy rose from 5% to 45.3% on algorithmic tasks and from 73% to 93.4% on GSM8K. In a setting where the model was trained on easy problems and applied to hard ones, MATH also rose from 28% to 44.6%. Even when trained only on synthetic verification rationales, it could catch subtle errors in math solutions.

In short, verifiers broadened from "classifiers that output a single score" to "models that give reasons along with their judgments." When a judgment comes with an explanation, humans can check the verifier's reasoning, and the explanation can also be fed back to the generator as revision instructions. However, because the judgment is generated as text, each verification also costs more. This cost becomes an important issue in the next thread.

Thread ④: Why the verifier becomes the bottleneck in inference-time compute

Large Language Monkeys paper treated "sampling many times" as another axis for scaling performance. Across multiple tasks and models, coverage (the fraction of problems solved by at least one of the samples) kept rising as the number of samples grew into the four-digit range. The relationship was often log-linear. In domains where answers can be checked automatically, such as coding or formal proofs, coverage translated directly into performance. On SWE-bench Lite, DeepSeek-Coder-V2-Instruct's solve rate was 15.9% with 1 sample and rose to 56% with 250 samples. This exceeded the single-sample state of the art at the time, 43%.

In domains without an automatic verifier, though, the story is different. According to Large Language Monkeys paper, majority voting and reward models, the common ways to pick one of the candidates, plateaued beyond a few hundred samples. They could not make full use of the larger sample budget. Having the correct answer somewhere in the pile of candidates does no good if it cannot be picked out. Sampling keeps improving as it scales, but the ability to choose does not keep up. This is why the verifier becomes the bottleneck.

The test-time compute study by Snell et al. paper compared two ways of spending inference-time compute. One searches guided by a step-level process verifier; the other has the model revise its answers to change the response distribution. Which works better varied greatly with problem difficulty. Using a "compute-optimal" strategy that allocates compute appropriately per problem improved efficiency by more than 4 times over Best-of-N. In compute-matched comparisons, a smaller model even outperformed a model 14 times larger, but only on problems where the small model achieved some degree of success.

Taken together, the two studies send a clear message. Inference-time compute is not just about sampling more. The quality and cost of the verifier, which decides what to pick and where to spend more compute, set the limit on overall performance. If verifier calls are expensive, verification costs grow along with the number of candidates sampled.

Thread ⑤: Training with rule-based verifiers (RLVR) and current directions

Verifiers are used not only to pick answers at inference time but also as training signals. RLVR (reinforcement learning with verifiable rewards) trains a model with reinforcement learning using outcomes that can be checked by rules, such as matching a math answer or passing unit tests, as the reward. As Let's Verify Step by Step paper noted, for problems whose answers can be checked automatically, outcome supervision can be provided without humans. Rule-based verifiers can therefore produce training signals cheaply and at scale. The resampling limits study paper also mentions the practice of selecting training data for reasoning models through rejection sampling (keeping only solutions that pass a verifier). Math-Shepherd paper, covered earlier, is a case where a learned PRM rather than rules was used as the PPO reward.

Tasks that cannot be checked by rules again require a learned verifier. SWE-Critix paper started from the observation that coding agent tasks often lack an oracle (a correctness judge) that defines success criteria. The general-purpose verifier this study trained takes a task and an execution trajectory (a record of hundreds of interactions with a code repository and tools), predicts whether the task was completed, and explains its reasoning in natural language. The authors trained 30B models by combining SFT (supervised fine-tuning) and reinforcement learning. On a held-out test set, classification accuracy rose from 57% for the base Qwen3-30B-A3B reasoning model to 66% after SFT and 78% after reinforcement learning.

Another direction is to fix and grow the verifier itself. The verifier co-evolution study "Who Verifies the Verifier?" paper addressed the problem of open-ended tasks with no fixed verifier. Using hand-written rubrics, or LLM judges that grade models similar to themselves, as-is tends to cause reward hacking and shared blind spots. So the study synthesized small, mostly deterministic defect detectors from failure cases and combined them into an inspectable verifier. Verifiers were selected not by agent scores but by how well they agreed with a reference set of ten items. On MBPP+, agreement on held-out data improved by 0.21 over the initial hand-built combination.

Limits of imperfect verifiers and practical patterns

If the verifier is wrong, spending more compute does not help. The resampling-limits study paper analyzed false positives (false positive: cases where the verifier passes a wrong answer). If the false-positive probability is not zero, drawing again does not reduce it, so accuracy has an upper bound regardless of the compute budget. On HumanEval and MBPP, where unit-test coverage is limited, a model's single-sample accuracy was strongly correlated with its false-positive rate. The study therefore concluded that no matter how many samples are drawn from a weak model, it cannot reach the single-sample accuracy of a sufficiently strong model. Because the cost of false positives outweighed the gains, the optimal number of attempts was often fewer than 10.

The problem grows when a verifier is used as a reward, because the generator learns what the verifier likes. The Prover-Verifier Games study paper showed that optimizing CoT solutions only for correctness can make the solutions hard to read and check. It therefore alternated training between a 'sneaky' generator, which produces wrong solutions that fool a small verifier, and a 'helpful' generator, which produces correct solutions the verifier accepts. As a result, both the verifier's robustness and the solutions' legibility improved. The finding of the verifier co-evolution study paper is more of a warning. When the safeguard of a reference set was removed, the verifier collapsed into an empty grader that passed anything. Yet the skills trained with that verifier looked just as good. The researchers concluded that downstream task scores alone cannot vouch for a self-evolved verifier.

With these limits in mind, practical patterns fall into four groups. First, sample-and-select (Best-of-N) is the simplest, but its accuracy is tied to the verifier's false positives, so do not increase the number of samples blindly. Second, in search, use a step-level verifier to prune branches at intermediate steps and save compute. As the test-time compute study paper showed, allocate compute differently depending on problem difficulty. Third, a generate–critique–revise loop feeds the verifier's explanation back to the generator so it can fix its answer. This fits well with verifiers that attach reasons to their judgments, as in the Generative Verifiers study paper.

Fourth, in coding agents, unit tests are the easiest rule-based verifier. But as the resampling-limits study paper showed, if test coverage is narrow, the tests themselves become a channel for false positives. For long work traces that are hard to check with tests alone, researchers are exploring supplementing them with trained verifiers such as SWE-Critix paper.

Further reading, and questions leading into Part 2

If you are new to this, we suggest starting with the GSM8K verifier study paper to grasp the basic setup of Best-of-N and verifiers, then Let's Verify Step by Step paper to learn the difference between ORM and PRM. Next, it helps to read Large Language Monkeys paper and the resampling-limits study paper together. They let you understand, as a pair, what becomes possible when you increase samples and the upper bound created by verifier false positives. The Generative Verifiers study paper and the test-time compute study paper are advanced material to read after that.

The question left at the end of this part is 'what makes a good verifier?' Judging from the studies covered so far, at least three properties seem necessary. The first is accuracy, especially the ability not to pass wrong answers. The second is cost. When candidates grow to hundreds or the verifier is called at every step of a search, the speed of a single verification dominates total compute. The third is calibration of scores. Only when a verifier that says '80% confident' is actually right about that often can you decide when to draw more, when to stop and when to hand off to a human.

Part 2 picks up this question. It looks at how using Jev, a decision model aimed at fast, calibrated judgments, as the verifier can improve the performance of a generator–verifier setup. It starts from two questions. When should you use long, explanatory judgments like those of generative verifiers, and when fast judgments? And how should confidence be used to reduce false positives?