Paper review · November 5, 2026
Think Less, Locally: Local Preference Learning for Reasoning Compression
The method finds the point in reasoning where overthinking begins and uses DPO to train the model to prefer a shorter correct continuation from that same point, greatly reducing CoT length while maintaining accuracy.
Problem
Large reasoning models keep deliberating even after they have obtained the information needed for the answer, which raises inference cost. Existing compression techniques compare entire trajectories or truncate parts of them. As a result, they cannot directly teach the local decision of "whether to reason further from the current state or wrap up here."
Method
LoCo builds training data in three steps. First, it splits the model's own CoT into paragraph-level steps and selects points with low step confidence and high variability as overthinking candidates. A judge model (Qwen3-30B) then re-checks whether the next step repeats content already verified. Second, it fixes that prefix, samples eight continuations, and picks the shortest correct one. Third, it groups (the same prefix, the short correct continuation, the original continuation) into preference pairs and trains with DPO. It was applied to DeepSeek-R1-Distill-Qwen-7B with LoRA, and the training data used only self-generated reasoning traces on math problems.
Results
- On five math benchmarks (MATH-500, AMC 2023, Olympiad, AIME 2024 and 2025), output length fell by 41.7–45.4%, and accuracy was maintained or improved.
- Despite training only on math data, output length fell by 23.4–36.3% on three code benchmarks (MBPP, LiveCodeBench v5, CodeContests). Among the compared methods, it had the highest AUC on all code benchmarks.
- The authors report that the improvement in the accuracy-efficiency trade-off was especially pronounced on difficult datasets that require both a high compression rate and high accuracy.
Limitations
Experiments were run only on one model, DeepSeek-R1-Distill-Qwen-7B. Overthinking detection requires a larger judge model, and continuation verification requires ground-truth labels. This limits extension to tasks where answers are hard to check. In the provided text, the detailed per-benchmark comparisons with baselines are cut off.
Why it matters
When serving reasoning models, token count translates directly into latency and cost. Rather than shortening outputs uniformly, LoCo teaches the model "when to stop." This makes it a practical post-training method that can lower cost while reducing the risk of losing performance on hard problems.
Key terms
- Overthinking
- A phenomenon in which a model keeps reasoning unnecessarily, re-examining the same content, even after it has enough information to give an answer.
- DPO(Direct Preference Optimization)
- A method that uses preference pairs (good response, bad response) to train a model directly to raise the probability of good responses, without a separate reward model.
- Reasoning state
- A prefix made by concatenating the prompt and the reasoning steps generated so far. It is the point at which the model decides whether to keep reasoning or reach a conclusion.
- CoT(Chain-of-Thought)
- An approach in which a model generates intermediate reasoning step by step before giving its final answer.
Original
Authors: Muyao Yuan, Liyan Xu, Weizhan Zhang, Muyan Jiao, Jiangnan Li, Jinchao Zhang, Jie Zhou · Posted 2026-11-03 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.