Paper review · November 6, 2026
What Does Multi-Turn Reinforcement Learning Actually Improve in LLM Agents?
What Does Multi-Turn Reinforcement Learning Really Improve in LLM Agents?
Using Pass@(K,T), which varies the number of samples and the interaction turn budget together, the authors show that gains from agentic reinforcement learning come from error recovery and long-horizon execution rather than per-step judgment, and are clear only when the turn budget is generous.
Problem
Multi-turn agentic reinforcement learning has improved performance on long-horizon tasks such as coding and tool use. But evaluation has mostly been limited to trajectory-level success rates, so it was unclear what actually improved and whether the gains hold in environments with tight turn budgets, as in real deployment.
Method
Using the Pass@(K,T) metric, which considers sampling width K and interaction depth T together, the authors compared base models and GRPO-trained models on DeepCoder-MT, BFCL, InterCode-SQL and FrozenLake. Step-level action quality was measured with exact Q-values computed by dynamic programming for FrozenLake and with unit-test pass rates for DeepCoder-MT. Each model sampled 16 actions from the same state for comparison, and "error states," in which an error was injected at the previous step, were evaluated separately. The mechanism was analyzed with action-level NLL and the cumulative trajectory log-likelihood ratio (LLR), and BudgetMix RL, which varies the turn budget during training, was proposed as an alternative.
Results
- Even as the sample count K increased, the reinforcement learning model consistently outperformed the base model. The authors report that, unlike single-turn reinforcement learning, it goes beyond the base model's Pass@K ceiling. However, at K=8, reducing the turn budget T shrank this advantage, and on some tasks it was reversed.
- From the same state, the reinforcement learning model chose the better action only 46.8% of the time on DeepCoder-MT and 42.8% on FrozenLake. In error states, by contrast, the figures were 67.3% and 65.7%, respectively. When continuing from error states, the reinforcement learning model also had higher final success rates: 51.3% vs. 38.6% on FrozenLake and 38.5% vs. 22.9% on DeepCoder-MT.
- On Terminal-Bench, the reinforcement learning model's actions were quite plausible even under the base model (average NLL 12.20 vs. 10.25). But the cumulative LLR of its successful trajectories was about 228, far larger than the 26 for the base model's successful trajectories.
- During DeepCoder-MT training, the trajectory reward rose by 0.331 while the step-level reward fell by 0.097. Exact process rewards (oracle Q-values, GiGPO) mitigated the problem on FrozenLake but had no effect on BFCL. The authors report that BudgetMix RL was robust across multiple budget conditions.
Limitations
The step-level quality analysis was limited to two tasks where exact evaluation is possible, FrozenLake and DeepCoder-MT, and mainly used Qwen3-4B with GRPO. The authors themselves describe their discussion of solutions as "preliminary," and specific figures for BudgetMix RL could not be confirmed within the provided text.
Why it matters
It shows that in real deployments with limits on turns or cost, or with irreversible actions (financial transactions, database operations), choosing a reinforcement learning agent based only on trajectory success rates can be risky. A practical implication is that agent evaluation should include a turn-budget axis and step-quality metrics.
Key terms
- Pass@(K,T)
- The probability that at least one succeeds when K trajectories are sampled and each is allowed up to T turns. It considers sampling width and interaction depth together.
- Turn budget (interaction budget)
- The maximum number of exchanges an agent can have with the environment on a single task.
- Error recovery
- The ability to see the result of a wrong action and correct it in the next step.
- Process reward
- A reward signal given at each intermediate step, not just for the final result. It is used to train step-level action quality directly.
Original
Authors: Kejian Zhu, Zhuoran Jin, Shangqing Tu, Tianyi Men, Yupu Hao, Jiachun Li, Kang Liu, Jun Zhao · Posted 2026-11-04 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.