Paper review · October 14, 2026
Thinking Inertia: LLMs That Keep Thinking Even When Told Not To
Thinking Inertia: LLMs Keep Thinking When Told Not To
Splitting responses into 'pre-answer text' and the 'final answer' and measuring them with a three-level metric showed that thinking-off settings and instructions could not reliably eliminate visible reasoning.
Problem
LLM 'thinking mode' has been studied extensively, but 'not thinking' has received little attention. Prior work judged it by surface features such as thinking mode being disabled, the absence of <think> tags, or short outputs. However, reasoning can appear even with thinking mode off, and long outputs may be filler rather than actual reasoning.
Method
Responses were split into pre-answer text (T) and the final answer (A) and evaluated at three levels. ① ETR (Empty-Thinking Rate): strictly checks whether only the answer was given. ② QRel: measures the relevance between the question and T using Qwen3-Embedding-4B. ③ EIR (Explicit Inference Rate): a blind GPT-5.5 judge determines whether T contains explicit reasoning steps. Six prompt interventions (Think-On/Off, CoT, Soft, Strict, Prefix) were applied to six LLMs and compared on Boolean, multiple-choice and open-ended tasks. The validity of the metrics was also checked with human evaluation.
Results
- Even with thinking mode turned off, DeepSeek-V4-Flash still showed explicit reasoning in 99.9% of its open-ended responses.
- The authors reported that visible reasoning depends more on the answer format (answer space) than on the switch setting. The reasoning rate was lowest for Boolean, intermediate for multiple-choice and highest for open-ended tasks ('Thinking Inertia').
- On open-ended tasks for five LLMs with thinking mode fully off, a strict 'answer only' instruction raised average ETR from nearly 0 to about 40%. In exchange, average accuracy dropped by about 15 percentage points.
- Rephrasing the same question in a format that shows candidate answers made answer-only responses much easier.
- Explicit reasoning labels from five human raters agreed with the GPT-5.5 judge 98.8% of the time on average.
Limitations
The authors state that they analyzed only visible responses and do not address latent reasoning inside the model. The study is limited to six models and six prompt interventions. EIR depends on an LLM judge (robustness was checked with a judge from a different model family).
Why it matters
For services that use 'reasoning off' because of latency or token cost, this shows that you should measure actual outputs rather than trust the setting alone. On open-ended tasks, there is also a trade-off: suppressing reasoning lowers accuracy.
Key terms
- Thinking Inertia
- The phenomenon where visible reasoning keeps appearing despite instructions or settings telling the model not to think. It gets stronger the more open-ended the answer format is.
- ETR(Empty-Thinking Rate)
- The share of responses that contain no text other than the final answer.
- LLM-as-judge
- An approach where another LLM is given a rubric and asked to evaluate outputs.
Original
Authors: Dianqiao Lei, Kevin Qinghong Lin, Pan Lu, Philip Torr, James Zou · Posted 2026-10-08 · License CC0 1.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.