Paper review · November 3, 2026
R²Orch: Recursively Improving Agent Orchestrators Through Self-play Inside a Skill Internalization Loop
R$^2$Orch: Recursive Self-Improvement for Agent Orchestration via Self-Play within a Skill-Internalization Loop
By layering a skill withdrawal loop on top of self-play between a Proposer that picks tasks and a Director that builds agent graphs, the authors trained an orchestrator that works without skills and outperformed baselines on 12 benchmarks.
Problem
Training orchestrators has three problems. (1) It depends on task pools (curricula) collected in advance by people. (2) In group-based reinforcement learning such as GRPO, if all Rollout results for a task are the same, the advantage becomes 0 (advantage collapse), and removing skills increases the number of such tasks again. (3) Skills are evaluated with a single noisy score to decide whether to keep or discard them, so the decisions are unstable.
Method
In the inner loop, the Proposer picks tasks, and the Director (the orchestrator) produces Trajectories that edit the agent graph, trained with GRPO. The outer loop shrinks the skill library. Recursive Learnability uses 4p(1-p) as the Proposer reward, where p is the success rate with the skill under review removed. As a result, tasks whose outcomes still vary without the skill are selected. Skill Withdrawal runs a controlled experiment in which the skill is hidden from most of each Rollout group and shown to only a few, and the skill is withdrawn when the Bayesian posterior of its effect indicates it is 'not needed'. Failures are distilled into skills, and skills are in turn internalized into the parameters.
Results
- Average accuracy across six in-distribution benchmarks is 89.79, higher than the strongest baselines PATS (86.00) and FlowSteer (82.34).
- It scores 81.33 on AIME 2026, higher than PATS (74.00), and its ALFWorld success rate is 90.63, higher than PATS (87.97).
- HotpotQA EM is 91.09, higher than PATS (89.06) and the Qwen3.5-9B backbone (57.81).
- The authors report that performance also improves on six other Worker backbones without training, and that accuracy holds when tested without skills.
Limitations
The Director in the experiments is based on a Qwen3.5-9B-class backbone, and the Worker pool is fixed. The theoretical results (Propositions 1 and 2) rely on assumptions such as a small trust region and two time scales. The OOD table is cut off in the text, so only part of it could be checked. Comparisons of training cost and number of Rollouts could not be found in the provided text.
Why it matters
It presents a training setup in which the orchestrator picks its own tasks and reduces its reliance on skills in the prompt, without people having to keep managing curricula and skills by hand. Since skills do not need to be put in the prompt at deployment, context costs also go down.
Key terms
- Advantage collapse
- In training that compares several Rollouts of the same task with each other, as GRPO does, this is when the learning signal disappears because the results are all successes or all failures.
- Self-play(Proposer–Director)
- A training setup in which one model picks the tasks to learn and another model solves them, improving both policies together.
- Recursive Learnability
- A metric that scores a task's learning value by computing 4p(1-p), where p is the success rate with the skill under review removed. It is largest when p is 0.5.
- Skill Withdrawal
- A procedure that measures a skill's effect by randomly splitting a Rollout group into with-skill and without-skill, and removes the skill from the library if it is judged to have no effect.
Original
Authors: Mingda Zhang, Yun Huang, Qiang Huang, Yanjin Li, Rui Qian, Qichao Zhao, Tiesunlong Shen, Na Zhao, et al. · Posted 2026-10-15 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.