Paper review · October 29, 2026
Multi-scale policy optimization (MPO) for long-horizon LLM agents
Multiscale Policy Optimization for Long-Horizon LLM Agents
By combining Advantage estimation that propagates credit at the turn level and refines it at the token level with turn-level importance sampling, the method consistently outperformed existing RL baselines in agent reinforcement learning.
Problem
LLM agents interact with the environment turn by turn, such as search queries or tool calls, but generate each action token by token. Critic-free methods such as GRPO share a single Advantage across an entire Rollout, so they cannot tell apart the contribution of each turn or token. Token-level methods such as PPO treat every token as an action and do not reflect turn boundaries.
Method
Analyzing critic values during search-agent training showed that the variance arose mainly between turns. On this basis, the authors treat each completed turn as a macro action and derive a turn-level improvement lower bound. They also show that, under the same KL constraint, the token surrogate incurs an extra penalty from distribution shift within a turn. The proposed MPO has two parts. (1) MAE: turn-level GAE propagates credit across turns, and then local traces within each turn are centered to zero mean and added. This assigns different credit to each token while preserving the turn average (the solution to a constrained least-squares projection). (2) The probability ratio of the whole turn is used as the importance weight, with stop-gradient preserving each token's own gradient, and updates use a clipped surrogate.
Results
- An analysis of 2,048 Trajectories and 16,323 turns from one search-agent training run found that, at the median, 89.5% of critic value variance was explained by differences between turn means.
- In 93.8% of Trajectories, between-turn variance exceeded within-turn variance.
- The authors ran experiments with 4B and 30B-A3B models on search, tool-use and automation tasks and reported consistently higher performance than strong RL baselines. Specific numbers are not in the provided text.
Limitations
Per-benchmark results are cut off in the provided text, so the size of the improvement cannot be checked. The theoretical analysis assumes a fixed turn horizon and a single final reward. A separate critic must be trained, so the training cost may be higher than for critic-free methods. Only two model sizes were tested.
Why it matters
The paper offers a principled way to split the unit of credit assignment into two levels, turns and tokens, when applying reinforcement learning to multi-turn tool-using agents. Its structure can be added fairly easily to PPO- or GRPO-style pipelines, so it has room for practical use.
Key terms
- Credit assignment
- The problem of deciding which actions or tokens in a Trajectory deserve credit for a single final reward.
- GAE
- Generalized Advantage Estimation. It estimates the Advantage by accumulating TD errors with exponential weights, and lets you trade off bias and variance.
- Turn-level importance ratio
- The ratio between the probabilities that the new policy and the old policy generate the full output of one turn. It is obtained by multiplying all the per-token ratios.
Original
Authors: Tao Ren, Hui Yang, Jia Deng, Tingyang Chen, Qiyao Wang, Shuo Tang, Ziyang Zeng, Yuhua Jiang, et al. · Posted 2026-10-26 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.