Reasoning Models and Test-Time Compute: Making Models Think Longer Before Answering

Why it was needed: scaling models was not enough

Large language models (LLMs) improved on many tasks as their parameter counts grew. But the CoT paper paper pointed out that scaling model size alone makes it hard to achieve high performance on problems that require multiple steps, such as arithmetic word problems, commonsense reasoning and symbolic manipulation. Standard few-shot prompting, which shows only a few question-answer pairs, did not work well on tasks that require reasoning, and in many cases it did not improve much even as models grew larger.

Another route was to train models directly on solution steps. But the same paper noted that producing high-quality solutions at scale costs far more than collecting simple input-output pairs. The problem came down to how to give the model 'room to think before it answers.'

This shifted attention from the training stage to the usage stage. What if, instead of having the model produce an answer in one go, it spent more computation at the moment it generates the answer (at inference time, that is, test time)? This is the idea behind test-time compute, and models that have absorbed it as a default habit are commonly called reasoning models.

Core idea: spend more time on hard problems

Think of someone taking an exam. They write down answers to easy questions right away; for hard ones they work through the solution on scratch paper, check their work, and if they get stuck, try a different approach. The test-time compute study paper also states that it starts from the observation that people improve their judgments by thinking longer on hard problems.

This analogy contains two elements. One is the 'scratch paper': CoT (Chain-of-Thought), in which the model generates intermediate reasoning steps as text. The CoT paper paper explained that going through intermediate steps lets multi-step problems be broken down, so more computation can be allocated to problems that need more steps. The other is 'time allocation.' Instead of spending the same time on every problem, the solver looks at the problem and decides how much more to think, and in what way.

Seen this way, a reasoning model is 'a model with an ingrained habit of using scratch paper,' and test-time compute is 'the strategy for deciding how much time to spend on that scratch paper, and how.' They are not separate technologies but two sides of the same idea.

How it works: revise sequentially, or sample in parallel and choose

The test-time compute study paper divided ways of scaling computation into two broad axes. The first is changing the distribution from which the model produces answers, with sequential revision as the main example. The model writes a first answer, looks at it and writes a revised answer, then revises again, continuing in a single chain. The second is using a verifier. The simplest form is best-of-N, which samples several answers at once (in parallel) and picks the highest-scoring one; another approach uses a process-based reward model (PRM) that scores each solution step and searches by following those scores.

The study's key observation is that which approach works better depends on problem difficulty. On easy problems, where the base model already produces plausible answers, sequential revision that refines the first answer step by step was more effective. On hard problems that require exploring multiple solution directions, it was better to sample new answers independently many times or to run a tree search guided by the PRM.

The study therefore proposed a 'compute-optimal' strategy that first estimates the difficulty of each question and chooses the computation method accordingly. Difficulty is measured from the base model's point of view, and this value was used to predict which computation method would work well. In short, the process is: receive the question → estimate difficulty → sequential revision or parallel sampling and verification → select the final answer.

How it developed: from prompting to compute allocation, and then reinforcement learning

The starting point is CoT prompting paper. This work showed that simply including examples made of three parts, <input, chain of thought, output>, in the prompt draws out reasoning ability in sufficiently large models. Showing PaLM 540B just 8 CoT examples achieved the then state-of-the-art accuracy on GSM8K, a benchmark of math word problems, beating even GPT-3 fine-tuned with a verifier. What mattered was that this was achieved through prompting alone, without retraining the model.

The second milestone is the study that systematically analyzed test-time compute paper. Earlier work had produced mixed results: some found inference-time computation helpful, while others found its benefits limited on complex tasks such as math. Using the compute-optimal strategy described above, this study improved test-time compute efficiency by more than 4 times over best-of-N. It also found that, in a FLOPs-matched comparison, on problems where a small base model achieves some degree of success, spending more inference-time computation beat a model 14 times larger. It put 'make the model bigger, or make it think more?' on the same scale.

The third is DeepSeek-R1 paper. This work reported that the reasoning ability of LLMs can be drawn out through pure reinforcement learning, without human-labeled reasoning trajectories (records of solution steps). During training, reasoning habits such as self-reflection, verification and switching strategies depending on the situation emerged on their own, and on tasks with checkable answers, such as math, coding competitions and STEM, the model outperformed models trained with supervised learning on human demonstrations. The authors also stated that the reasoning patterns that emerged this way can be used to improve the reasoning ability of smaller models. If CoT was about 'making the model think by showing examples,' R1 went a step further by 'training the habit of thinking itself.'

Where it is heading: from how much to where

Rather than simply increasing computation, recent research is moving toward deciding more precisely when and where to use it. A study on a confidence-calibrated controller paper treats each step of verifier-guided reasoning as a three-way choice: trust the current answer, produce an alternative answer, or spend more compute. By predicting the success probability of each choice and assigning a cost to computation when choosing, it improved utility over fixed policies while cutting online computation by about 30% on width-scaling tasks and about 16% on revision tasks. However, the authors stated that the utility differences against other baselines were small and mixed, and concluded that this should be seen not as 'a stronger reasoner' but as a resource allocation problem under uncertainty.

REWIND paper focuses on 'where to start regenerating.' Motivated by the problem that a single early mistake in reasoning locks in everything that follows, the authors analyzed Qwen2.5-Math-7B on MATH500 and found that wrong answers had much higher average token entropy (how hesitant the model is about the next token) than correct ones, with an average relative increase of about 112% within the same problem. Based on this, they built a controller that requires no additional training and regenerates only the part after points where entropy changes sharply; across three model families and five benchmarks, it scored 0.51–10.00%p higher than the strongest baseline.

How budgets are 'measured' has also become an issue. A study on budget boundary effects paper replayed 19,200 solution traces from 120 problems in AIME, BrUMO and HMMT, comparing stopping immediately when the token cap is reached with letting an in-progress attempt finish. Even under the same cap, accuracy and actual cost differed depending on the stopping rule, and securing more candidate answers did not guarantee higher accuracy. The study recommended that reports of budget curves state the cap, the actual cost spent, the candidates eligible for selection, the stopping rule and the answer-selection method together.

Limitations and open issues

First, test-time compute is not a cure-all. Even in the test-time compute study paper, inference-time computation beat a larger model only on problems where the small model already achieved some degree of success. When the number of inference tokens grew large relative to pretraining tokens, hard problems were better served by spending compute on pretraining instead. Problems where the model cannot find a foothold in the first place do not get solved much better by thinking longer.

Second, it relies heavily on verification signals. Parallel sampling is only meaningful when there is a verifier or reward model to pick out the good answer, and the reinforcement learning in DeepSeek-R1 paper also showed its strength on tasks where the correct answer can be checked, such as math and coding. Whether the same effect holds for open-ended questions without a single correct answer needs to be examined separately. The same study paper also reported that bringing out revision and verification abilities required fine-tuning tailored to those abilities.

Third, there is cost and latency. Making a model think longer increases tokens and slows responses. That is why recent studies are interested in controllers that reduce compute paper and methods that regenerate only the parts that need it paper. And as the budget-boundary effect study paper shows, the way cost and accuracy are compared can itself change the conclusion, so the measurement conditions should be checked when reading the numbers.

Fourth, the thinking process is hard to take at face value. The CoT paper paper says the reasoning process can serve as a window into the model's behavior, but adds the caveat that fully characterizing the model's internal computations that support an answer remains an open question. Even if a written solution looks plausible, one cannot conclude that it matches the actual basis for the decision.

Further study

If you are new to this, start with the CoT paper paper. The examples in Figures 1 and 3 alone make the idea of 'having the model write out intermediate steps' click, and the four advantages of CoT summarized in Section 2 became the starting point for later research.

Next is the test-time compute study paper. The parts that experiment separately with sequential revision and PRM search, and the part that chooses a compute strategy based on difficulty, form the basic vocabulary of this field. The figure in Section 1 comparing a small model and a large model at matched FLOPs is especially helpful for understanding the trade-off between 'model size and thinking time.' Reading DeepSeek-R1 paper after that shows how such reasoning habits are instilled into a model through reinforcement learning.

To follow recent developments, read in order REWIND paper for 'finding where to fix using uncertainty signals,' the calibrated-confidence controller study paper for 'deciding whether to spend more compute,' and the budget-boundary effect study paper for 'reporting budget experiments fairly.' All three address the same question from different angles: allocating compute well matters more than simply spending a lot of it.