Paper review · October 16, 2026
Do More Methods Make Better Agents? Revisiting Self-Distillation Methods for Agentic Reinforcement Learning
More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning
It compared 10 self-distillation-based agentic reinforcement learning methods against GRPO under the same conditions and found that none consistently beats GRPO.
Problem
Self-distillation methods for agentic reinforcement learning are multiplying quickly. But each paper reuses baseline scores from different experimental conditions, and contemporary methods are rarely compared directly. Training stability and compute cost are also seldom reported, making it hard to tell where the reported gains come from.
Method
Ten representative methods, together with GRPO, were trained and evaluated under the same conditions on four environments (ALFWorld, ScienceWorld, Search-QA and WebShop) at three model sizes. The pretrained model and all 11 trained methods were evaluated repeatedly with 8 rollout seeds. Beyond final performance, the study also analyzed training progress, stability and actual measured compute cost.
Results
- The comparison involved 132 training runs in total, amounting to 12,124.4 H800 GPU hours.
- The authors report that gains over GRPO appeared only in some settings, and no method consistently beat GRPO across all settings.
- In some cases, training was unstable even when final performance was good.
- Spending more on training did not consistently improve performance.
Limitations
This analysis is based on the abstract. Per-method numbers, model sizes and the detailed list of compared methods could not be checked. Conclusions are limited to the four environments and three model sizes.
Why it matters
Agent training is expensive, so without a fair common comparison standard it is easy to pick the wrong method. It shows that before adopting a new technique, teams should weigh it against a simple baseline such as GRPO, including stability and cost.
Key terms
- Self-Distillation
- An approach in which a model trains itself using its own outputs, or its own predictions under better conditions, as the teacher signal.
- GRPO
- A reinforcement learning algorithm that samples multiple responses for the same prompt and computes the advantage from relative rewards within the group. It does not need a separate value model.
- Rollout Seed
- A value that sets the randomness of agent runs during evaluation. Repeating evaluation with multiple seeds shows how much results vary.
Original
Authors: Linhao Zhong, Bozhen Fang, Xuehao Wang, Duochao Shi, Zekai Luo, Guangming Yao, Yi Yuan, Hao Chen, et al. · Posted 2026-10-14 · License arXiv 비독점 배포 라이선스 · Based on the abstract
This review is an AI-written summary of the paper. Please treat the original paper as the reference.