Deep dives · 2026-11-16 · Basic · By bombong
Controlling Reasoning Effort: Thinking-Token Budgets and Reducing Overthinking
Reasoning models keep re-checking their work even after finding the right answer. This overthinking wastes tokens and time, and sometimes overwrites a correct answer with a wrong one. This article lays out what thinking-token budgets, stop signals, step-level rewards and routing on whether to think each gain and lose.
Contents
- Why it was needed: the downside of "longer thinking is better"
- Key idea: thinking needs a budget and a brake
- How it works: stopping, teaching and rewarding
- How it developed: from cutting length to assigning step-level credit
- Where it is heading: trimming after the answer settles, and choosing whether to think first
- Limitations and open issues: look at lost correct answers before saved tokens
- Further study
- Overthinking is not just a cost problem. If a model keeps thinking after finding a correct intermediate answer and then overwrites it with a wrong one, accuracy drops too.
- A fixed thinking-token budget is simple and makes costs predictable, but it ignores the fact that different problems need different amounts of thinking.
- Research has moved away from penalizing total length uniformly and toward cutting only the steps or tokens where redundancy begins.
- Recent work goes a step beyond "how much to think" and chooses "whether to think at all" and "which behaviors to reduce."
- In practice, check first that no correct answers were lost on hard problems before looking at token savings. The goal is not to make thinking shorter but to make the model think as much as it needs to.
Why it was needed: the downside of "longer thinking is better"
Reasoning models write out a long intermediate solution, called CoT (Chain-of-Thought), before giving an answer. Spending more computation at answer time like this is called test-time compute. This approach has raised performance on math, science and tool-use problems. It also has a cost. Thinking tokens must be generated one at a time and are billed per token, so longer thinking raises both cost and response time.
Not all of the extra thinking is useful, either. Repeatedly re-checking and re-deriving after already reaching the correct answer is called overthinking. The ThinkBrake paper authors ran an "oracle stopping" experiment: they tried inserting the end-of-thinking marker (</think>) at the end of every sentence, then looked at the results afterward and picked the best stopping point. Thinking tokens fell by 72% and average accuracy actually rose by 8%. This means models fairly often find the right answer midway, keep thinking, and then overwrite it with a wrong one.
Concise-9B paper forced Qwen3.5-9B to produce an answer every 2,048 tokens while it was thinking. On sets of hard problems, 54–64% of thinking tokens came after "the point from which every forced answer matches the final answer." In other words, the model kept thinking after its answer had already settled. A similar pattern shows up in tasks that include images. REChart paper reported an inverted-U relationship between reasoning length and performance in chart editing: when thinking runs too long, the model makes up details that are not in the image (hallucination) or circles through the same reasoning.
There is also failure in the opposite direction. ReBalance paper points to underthinking, where a model fails to explore enough despite having the ability, alongside overthinking, which wastes computation steps on easy problems. Existing methods that suppress reflective phrases or shorten length can end up causing underthinking. So the goal in this field is not "shorter" but "as much as needed."
The scale of overthinking
Key idea: thinking needs a budget and a brake
Think of a student taking an exam. A student who checks easy problems five times runs out of time, and a student who rushes through hard problems gets them wrong. A student who does well looks at each problem and estimates how much time to spend, then puts the pen down once they are confident it is done. Controlling reasoning effort means giving the model these two senses: a sense of budget and a sense of when to stop.
The simplest tool is a thinking-token budget. You set a cap on the number of tokens used for thinking and cut thinking off when it is exceeded. It is easy to implement and makes costs predictable, but it ignores the fact that different problems need different amounts of thinking. The VSRM paper authors noted that most existing efficiency methods must accurately judge task difficulty in advance to set a budget or choose a reasoning mode, which reduces their flexibility and reliability.
Research therefore splits into three main questions. The first is the brake: deciding "when to stop" during thinking. The second is pruning: finding "which parts are redundant" within long thinking. The third is routing: choosing before starting "whether to think, and how much." The three approaches are not mutually exclusive and can be used together.
How it works: stopping, teaching and rewarding
A reasoning model writes its thinking section sentence by sentence, then emits the end-of-thinking marker and writes the answer. Effort control happens at several points in this loop. The first is inference-time control. The model is left unchanged, and signals that appear during generation are monitored. ThinkBrake paper measures, at the end of every sentence, the gap in log probability between "the most likely next token" and "</think>". When this gap narrows, meaning the model itself starts to treat ending as reasonable, thinking stops. It requires no training and cut thinking tokens by up to 30% on math, science question-answering and tool-use benchmarks. ReBalance paper uses the model's confidence as a continuous signal. Large swings in confidence are read as overthinking, and persistent overconfidence as underthinking. It then adjusts, in real time, the strength and direction of a steering vector (a value that pushes the hidden representation in a particular direction) built from a small dataset, cutting redundancy and increasing exploration when needed.
The second is changing habits through training. Self-Braking Tuning paper aims to have the model stop on its own without external mechanisms. It builds a metric that identifies overthinking relative to the correct answer and uses it to find unnecessary steps in solutions. Based on this, it creates training data whose length varies by problem, along with "brake prompts," so the model learns to end its thinking at the right point.
The third is shaping reinforcement learning rewards. The most common method adds a length penalty to the correctness reward. However, ATTNPO paper points out that a length penalty applied uniformly across the whole trajectory does not reduce length well and also lowers accuracy. Because it treats every step the same, it cannot tell necessary steps from redundant ones. So research moved toward assigning rewards and penalties to each step separately. VSRM paper rewards each step based on the outcome of intermediate states. ATTNPO divides credit among steps using the scores of specific attention heads that focus on essential steps.
These designs make different trade-offs. Inference-time control has no training cost and can be attached directly to existing models. In exchange, it adds decision cost at every generation and requires access to the model's internal signals. Training-based methods pay the training cost once and then need no extra mechanism at inference. In exchange, their behavior must be re-checked on problem types that differ from the training data.
How it developed: from cutting length to assigning step-level credit
In 2025, Self-Braking Tuning paper moved away from earlier approaches that relied on external control and opened the direction of "the model applying the brake itself." VSRM paper, instead of setting a budget in advance, held that the key is to encourage effective steps and penalize ineffective ones within reinforcement learning. It combined rule-based step-level rewards grounded in the outcomes of intermediate states with PPO and Reinforce++, greatly reducing output length while maintaining performance on AIME24 and AIME25. ThinkBrake paper, by contrast, decided when to stop at the decoding stage without training.
In 2026, work followed that pinpoints more precisely which steps and tokens are redundant. ATTNPO paper cheaply assigned step-level credit using attention signals inside the model, shortening reasoning while also improving performance on 9 benchmarks. ReBalance paper reduced redundancy and raised accuracy without training across four models from 0.5B to 32B and nine benchmarks. REChart paper extended the scope to chart editing. It designed an "efficiency reward" that gives each rollout a random thinking budget, cuts reasoning off at that point, and rewards the final reasoning segment in proportion to its contribution to the result.
AES paper argued that applying a length penalty uniformly across the whole sequence causes "credit confusion," in which essential steps cannot be distinguished from redundant ones. As a proxy for this, it focused on tokens with high entropy (uncertainty). Such tokens often coincide with turning points where reasoning goes back to verification. AES uses the length of efficient answers from the same group as a reference, keeping high-entropy tokens that appear before that length and penalizing high-entropy tokens that appear after it. As a result, it cut reasoning length by up to 52.9% while raising average accuracy by up to 3.8% across eight benchmarks. In short, the trend has moved from "cut total length" to "find where redundancy begins and cut only that part."
Where it is heading: trimming after the answer settles, and choosing whether to think first
Concise-9B paper removes redundancy through simple self-training. It ran two rounds of on-policy rejection sampling fine-tuning, keeping only the shortest correct solutions the model generated itself and fine-tuning on them again. On 8,983 test problems with a 32K thinking budget, it used only 0.66 times the tokens of the original model while keeping accuracy the same within ±1 point. Of the thinking tokens saved, 86% came from the stretch after the answer had already settled. One point is worth noting: applying a simple rule such as "stop when the forced answer is the same three times in a row" to the original model lowered accuracy by 2.4–9.2 points. This is because the point where the answer settles can only be known precisely in hindsight.
HDM paper asks not "how much to think" but "whether to skip thinking entirely or think all the way." A single fixed model uses both modes. The fast mode produces no thinking tokens and answers within a single forward pass (one computation that passes the input through the model). The slow mode performs full reasoning. A lightweight self-routing block estimates the "value of thinking" and runs the slow mode only when that value exceeds a price λ set per request. On the agent task MiniWoB, HDM-27B cut token costs by 80.1%. Because the price can be changed per request, this approach treats reasoning effort like a service setting.
Routes of Reasoning paper targets "behaviors" rather than length. It starts from the problem that existing methods such as token budgets and steering only adjust the length of thinking and cannot choose which behaviors to reduce. The authors showed that router scores alone in an MoE (mixture of experts) model can predict six reasoning behaviors well: planning, execution, restatement, facts, reflection and conclusion. They then modified router scores during inference to suppress overthinking, reducing generated tokens by 30.4% at the median and up to 63.3%.
Limitations and open issues: look at lost correct answers before saved tokens
First, the cost of shorter thinking shows up on hard problems. When Concise-9B paper was allowed to think up to 80K tokens, it solved 25 fewer of 1,276 hard problems than the original model, most of them competition math problems. Even with the same average accuracy, there can be losses on hard problems at the tail of the distribution. The underthinking that ReBalance paper warned about is part of the same picture.
Second, fixed budgets are simple but crude. Cutting by budget makes costs predictable, but VSRM's paper point that you need to know the required amount of thinking in advance still stands. The large gain shown by ThinkBrake's paper oracle experiment is only an upper bound for the case where "the optimal point is known after the fact." The savings achieved by the actual method were smaller, up to 30%.
Third, each method requires different access. ThinkBrake needs access to token log probabilities, ReBalance to the hidden state, and Routes of Reasoning paper to the MoE router. Teams that use models only through APIs will find it hard to attach such inference-time control themselves, while teams serving open-weight models directly have more options. Training-based methods such as Self-Braking Tuning, VSRM and AES require a training pipeline and data with verifiable answers.
Practical criteria are as follows. Do not look only at token savings; look at accuracy by difficulty level as well. If easy requests are common and cost and response time matter, routing that chooses "whether to think" or a low budget fits. If hard problems make up a large share, choose a method that keeps a generous budget and trims only redundancy. When moving results validated on math to other domains, measure again. As REChart paper shows, in multimodal tasks overthinking can lead to hallucination, so it needs separate checking.
Checklist before tuning reasoning effort
Measurement
- Do you break down accuracy by difficulty?
- Do you count correct answers missed on hard problems?
- Do you measure response time as well as tokens?
- Have you re-measured outside the math domain?
Choosing a method
- Do you have access to log probabilities or internal states?
- Do you have data with verifiable correct answers?
- Is the share of easy requests large enough that routing makes sense?
- If multimodal, have you checked for increased hallucination?
Further study
To get a feel for the phenomenon first, read ThinkBrake's paper oracle stopping experiment and Concise-9B's paper measurement of "thinking tokens after the answer settles." Together, the two studies show how common overthinking is and why it is hard to reduce in practice.
It helps to read the methods by family. For training-free brakes, see ThinkBrake and ReBalance paper; for training the model to stop on its own, see Self-Braking Tuning paper. For reinforcement learning reward design, reading VSRM paper, ATTNPO paper and AES paper in that order lets you follow how "step-level rewards and penalties" became more precise. On the multimodal side, REChart paper is the starting point.
For recent directions, compare HDM's paper "whether to think" routing with Routes of Reasoning's paper "behavior-level control" to see what comes after length control.
Related terms
Reasoning model Test-time compute CoT RLVR Reinforcement learning MoE Token-based pricing Speculative decoding
References
Foundational papers
- ThinkBrake: Efficient Reasoning via Log-Probability Margin Guided Decoding (2025)
- Let LRMs Break Free from Overthinking via Self-Braking Tuning (2025)
- Promoting Efficient Reasoning with Verifiable Stepwise Reward (2025)
- ATTNPO: Attention-Guided Process Supervision for Efficient Reasoning (2026)
- Efficient Reasoning with Balanced Thinking (2026)
- REChart: Reasoning-Efficient Chart Editing with Large Reasoning Models (2026)
- Adaptive Entropy-Sparing for Efficient Reasoning (2026)
Recent research
Written by AI from the listed references and reviewed. Please check the original sources for details.