Paper review · November 13, 2026
Eliciting LLM self-learning ability with reinforcement learning
Incentivizing Self-Learning in Large Language Models via Reinforcement Learning
Training an LLM with reinforcement learning and fine-grained rewards to solve proof problems by navigating textbooks itself raised proof accuracy from 51.5% to 79.5%.
Problem
Solving problems in an unfamiliar field takes more than access to relevant information. The model has to understand new principles and work them into its solution. Existing RAG and learning from pre-bundled context do not handle well the interplay between knowledge acquisition and reasoning. The authors study 'self-learning': the ability to find, study and apply what is needed from long reference material on one's own.
Method
5,183 exercises were drawn from 158 math textbooks to build 'problem–textbook' pairs, with solutions removed from the textbooks visible to the model. The BookNavigator interface offers three actions: viewing the table of contents (get_toc), keyword search (search_index) and reading a section (read_section). Intern-S2-Preview-35B was trained with GRPO for 50 steps, with two rewards added to proof correctness (passing requires GLM-5.2 to judge it correct in all 8 checks). (1) An exploration bonus given when tools are used in a correct Trajectory, decayed from 1 to 0.25 on a cosine schedule. (2) A frozen copy of the initial policy rates tool use in each round as 'helpful, neutral or harmful', and the advantage is redistributed per round accordingly.
Results
- Overall proof accuracy rose 28.0 percentage points, from 51.5% to 79.5%. This was 6.4 percentage points higher than Top-k Sections (73.1%), the best fixed-retrieval baseline trained with the same RL method.
- Before training, Self-Learning (51.5%) scored lower than No-Book (55.7%), which solves without a textbook, but after RL it beat No-Book (68.5%) by 11.0 percentage points.
- It scored 85.2% on textbooks not used in training (OOD-Book) and 86.7% on a subject absent from training (probability and statistics, OOD-Subject), the highest of all compared methods.
- On BrowseComp-Plus and CL-bench, evaluated in their original form without BookNavigator, it also improved over the base model by 1.7 and 2.3 percentage points, respectively.
- The authors reported that using only correctness as the reward raises accuracy but reduces textbook exploration. Using the exploration bonus and tool credit together gave better performance in other domains without maximizing the number of tool calls.
Limitations
Both proof correctness judgments and tool-use ratings are left to LLM judges, so grading errors may creep in. The RL experiments involve a single model, Intern-S2-Preview-35B, at a scale of 50 rollout steps. The task is limited to proofs based on math textbooks, and textbook search is keyword-based. Gains on non-math tasks are modest, at around 1–2 percentage points. The input text was cut off partway, so the analysis of what drives the performance gains and detailed ablation numbers could not be checked.
Why it matters
It points to a direction of teaching models 'how to find and learn what they don't know on their own' rather than packing in more knowledge. When designing agents that solve problems while reading long material such as internal documents or manuals, it provides evidence that active exploration may work better than RAG that attaches fixed retrieval results.
Key terms
- GRPO(Group Relative Policy Optimization)
- A reinforcement learning algorithm that samples multiple responses to the same problem and updates the policy using, as the advantage, how much better each response is relative to others in the group.
- Self-learning
- The ability of a model, while solving a problem, to find, read, understand and apply the knowledge it needs from reference material on its own.
- Credit assignment
- The problem of deciding how much of the final reward to assign to individual actions within a Trajectory (here, tool calls in each round).
- LLM-as-judge
- An approach in which another LLM judges whether a model's output (here, a proof) is correct and uses that as an evaluation or reward signal.
Original
Authors: Lingkai Kong, Yuzhe Gu, Haiteng Zhao, Zijian Wu, Youqing Fang, Zhouqi Hua, Yanan Sun, Kai Chen, et al. · Posted 2026-10-29 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.