← Deep dives

Deep dives · 2026-11-12 · Advanced · By bombong

Long-horizon agent reinforcement learning: multi-turn rollouts and credit assignment

When you train an agent that calls tools dozens of times with reinforcement learning, the hardest part is deciding which turn deserves credit for a single final success or failure. This article covers three lines of solutions — branch comparison, value estimation by a reference model, and privileged self-distillation — along with current issues: turn budgets, tool credit and rollout cost.

Contents
Key points
  • Multi-turn agent reinforcement learning usually splits a single outcome reward at the end of the trajectory evenly across all turns, so even good actions inside a failed rollout get penalized.
  • Credit assignment solutions fall into three broad lines: branching from the same point and comparing outcomes (BranPO), building per-turn rewards from changes in a reference model's estimate of how likely the correct answer is (TRACE), and mixing in token- or turn-level signals from a teacher that has privileged information (SDAR, ADRS, AgentOPSD).
  • All three lines try to avoid training a separate value function (critic). They differ in what they need instead: extra sampling, ground-truth labels, or privileged information such as skills.
  • The gains from multi-turn reinforcement learning come less from better single-step decisions and more from error recovery and endurance on long tasks, and they shrink sharply when the turn budget given during training is reduced.
  • Recently, two new challenges have emerged: tool-call tokens receiving negative credit, and the memory and latency costs of running many task environments at the same time.

Why it was needed: judging dozens of turns by a single outcome

To solve a single question, an LLM agent goes back and forth over many turns: it searches, reads the results, searches again, runs code, and so on. A record of one full run of this process is called a rollout, and the sequence of turns within it is called a trajectory. Reinforcement learning has become the main method for post-training such agents, but the reward is usually just whether the final answer was correct. SDAR paper points out that this trajectory-level reward provides only 'coarse supervision' for long interactions.

The problem is that this single outcome is split evenly across every decision. BranPO paper describes this as 'a single outcome being assigned uniformly to all decisions.' TRACE paper argues that when a trajectory stretches to tens or hundreds of tool calls, the outcome reward becomes sparse and high-variance, and can even be misleading. A failed rollout often contains many useful actions that moved toward the goal, but outcome-only training gives those actions the same negative advantage (a value indicating how much better than average something was) as the final mistake.

On top of this comes a property BranPO paper calls 'non-monotonic correctness.' An early mistake may be corrected later, and an intermediate state that looked good may collapse after a later wrong decision. So judging that 'the current state looks good' is not enough to tell what action to take in each state. Credit assignment (distributing credit or blame for an outcome across individual decisions), the subject of this article, is an attempt to fill this gap.

Core idea: evaluate each play, not just the final score

A football (soccer) match is a useful analogy. If a team loses 2–1 and every pass by every player that day is rated equally bad, there is nothing to fix for the next match. A good coach goes through the match footage play by play and assigns credit and blame: 'this pass started the goal against us,' 'this defensive move stopped a threat.' Credit assignment in long-horizon agent reinforcement learning does the same thing. It keeps the final success or failure — the match result — but turns it into dense turn- or token-level signals that point out which decisions decided the outcome.

There are two things this tries to avoid. One is training a new value function (critic) model that separately predicts the value of each state. Training one more model costs resources, and if its predictions are wrong, the signal gets contaminated. The other is expanding large numbers of branches like a tree search, or having people label every step; BranPO paper notes that such existing approaches are expensive or produce noisy signals. So most research in this area looks for 'ways to extract play-by-play evaluation from what already exists, without training an extra model.'

The paths diverge depending on what is used as 'what already exists': replaying the match from the same play, asking a commentator who knows the answer for the odds at each play, or having a coach who has seen the playbook (privileged information) re-score the same play. The next section looks at how these three actually work.

How it works: multi-turn rollouts and three ways to build signals

The basic loop is simple. Given a task, the policy model (the LLM being trained) thinks, calls a tool, reads the observation returned by the environment, and chooses its next action. When this loop ends, the whole trajectory is recorded and an outcome reward is assigned based on the final answer. GRPO-family methods, which the SDAR, AgentOPSD and TACA papers use as baselines, run several rollouts on the same task and push up the rollouts that did relatively well within the group. Credit assignment research inserts a 'build per-turn signals' step between this outcome reward and the policy update.

The first line is branch comparison. BranPO paper cuts a trajectory at an intermediate point and resamples what follows, creating branches that share the same prefix but end with different outcomes. Since the prefix is identical, the difference in outcome can be attributed to decisions made after the branch point. On top of this, it adds sampling that allocates more branches to difficult points, and Redundant Step Masking, which prevents updates on duplicated steps.

The second line is the change in value as judged by a reference model. TRACE paper splits a rollout at each tool-call boundary, treats it as a sequence of state transitions, and asks a frozen reference model for 'the log probability that the correct answer comes out of the state so far.' It converts this into a state value and uses how much the value changed before and after an action (temporal difference, TD) as that action's reward. No separate critic or process-label training is needed, and needlessly repeated tool calls do not accumulate reward because their value changes cancel out. The trade-off is that it requires a gold answer.

The third line is privileged self-distillation. A teacher branch with privileged information available only during training — such as task-specific skills (procedural guidance) — re-scores the same tokens that were generated without the skills, providing dense signals. SDAR paper, ADRS paper and AgentOPSD paper follow this path, and all of them run rollouts and actual inference without skills. The differences among the three are covered in the next section.

The multi-turn rollout and credit assignment loopTask → Policy model; Policy model → Tools & environment(Tool call); Tools & environment → Policy model(Observation); Tools & environment → Trajectory(Record); Trajectory → Outcome reward; Outcome reward → Build per-turn signals(Single outcome); Branch comparison → Build per-turn signals; Reference model value change → Build per-turn signals; Privileged teacher scoring → Build per-turn signals; Build per-turn signals → Policy update(Per-turn signals); Policy update → Policy model(Update)TaskPolicy modelLLM being trainedTools &environmentSearch & codeexecutionTrajectoryFull turn recordOutcome rewardSinglesuccess/failurePolicy updateBuild per-turn signalsCredit assignmentBranch comparisonBranPOReference modelvalue changeTRACEPrivileged teacherscoringSDAR·ADRS·AgentOPSDTool callObservationRecordSingle outcomePer-turn signalsUpdate
The multi-turn rollout and credit assignment loop The upper part is the multi-turn loop between the policy model and the tool environment; the lower part is the step that turns a single outcome reward into per-turn signals. The dashed boxes are the three lines of methods for building those signals.

How the field developed: what each starting-point paper added

Two studies in February 2026 tackled the problem of breaking up the outcome reward head-on. BranPO paper, using the contrastive branching described above, created local contrastive signals without a value function, outperformed a range of baselines on several multi-hop QA (question answering that requires multi-step retrieval) benchmarks, and reported no additional training cost. In the same month, BAO paper focused on 'behavior between turns' rather than credit assignment. Proactive agents that ask the user questions and plan first face a Pareto frontier between task performance and user burden (asking too many questions is annoying). BAO reinforced information-gathering behavior and suppressed inefficient or redundant interactions, and reported both higher performance and less user effort on the UserRL benchmark.

Around the same time, WideSeek-R1 paper opened a different axis. Instead of 'depth scaling,' where one agent carries on over many turns, it used multi-agent reinforcement learning to train 'width scaling,' where a lead agent directs multiple subagents in parallel. The lead and subagents share a single LLM but keep separate contexts, and were jointly optimized on 20,000 broad information-seeking tasks. As a result, a 4B model reached 40.0% item F1 on the WideSearch benchmark, comparable to single-agent DeepSeek-R1-671B, and performance reportedly rose steadily as the number of subagents increased.

From May onward, privileged self-distillation formed its own line of work. SDAR paper found that porting on-policy self-distillation (OPSD) directly to multi-turn agents becomes unstable as turns accumulate, and that incomplete skill retrieval and use can cause the teacher to wrongly reject tokens. So it made reinforcement learning the main objective and attached OPSD as an auxiliary objective with a sigmoid gate. Tokens the teacher supports are distilled strongly, while tokens the teacher rejects are softly down-weighted. It reported gains over GRPO of +9.4% (ALFWorld), +10.2% (WebShop accuracy) and +7.0% (Search-QA), and avoided the instability of a naive GRPO+OPSD combination. In August, ADRS paper normalized teacher scores within each turn and used a TVA gate, which checks how closely the teacher's confidence tracks the actual reward, so that the signal is used 'only when that preference is relevant to the outcome.' In the same month, AgentOPSD paper aggregated token-level teacher–student log-probability differences into turn-level evidence, recursively updated a Bayesian belief at each turn, and identified turns where the belief shifted sharply as decisive turns. It needs neither a critic nor extra rollouts, and reported an 89.1% success rate on ALFWorld with Qwen2.5-7B.

TRACE paper, from July, builds per-turn rewards from TD changes in a reference model's value. It reported raising BrowseComp-Plus scores from 7.2 to 35.6 on Qwen3-4B and from 8.4 to 42.6 on Qwen3-30B-A3B with pure reinforcement learning alone, without an SFT cold start, agent-specific mid-training, or live web data.

BrowseComp-Plus scores before and after TRACE training

Before trainingAfter TRACE training

Qwen3-4B+28.4points
Qwen3-30B-A3B+34.2points
Figures reported in the TRACE paper, comparing each model before and after training with pure reinforcement learning only.

Comparing the three lines: what each trades off

All three methods share the property of not training a separate critic. But dense signals don't come for free, so each pays a different price. Branch comparison (BranPO) has to resample from intermediate points, which increases the number of rollouts. To keep it efficient, it concentrates branches on difficult points and adds a mechanism to mask duplicated steps. In return, it has the advantage of working with outcome rewards alone, without ground-truth labels or privileged information.

TRACE paper-style value change needs no extra rollouts; it only computes log probabilities from a frozen reference model. The cost is that it requires a correct-answer string, and signal quality depends on how well the reference model can tell that 'the agent is getting closer to the answer.' It suits tasks with a clear correct answer, such as search, and is hard to apply as-is to tasks without a single defined answer.

Privileged self-distillation reuses the same policy model as the teacher, so its strength is that it adds no new model and needs no extra rollouts. The cost is that privileged information such as task-specific skills must be prepared, and as SDAR paper pointed out, if those skills are retrieved or used incorrectly, the teacher may reject tokens for the wrong reasons. SDAR's asymmetric gate, ADRS paper's reward-relevance gate and AgentOPSD paper's turn-level belief updates can all be read as mechanisms for adjusting 'how much to trust this teacher signal.'

Trade-offs among three approaches to reward assignment

Branch comparisonBranPO

  • Works with outcome rewards alone
  • Requires resampling from intermediate points
  • Concentrates branches at difficult points

Value changeTRACE

  • No extra rollouts needed
  • Requires a ground-truth answer string
  • Redundant tool calls cancel out

Privileged self-distillationSDAR·ADRS·AgentOPSD

  • Reuses the same model as the teacher
  • Requires task-specific skills to be prepared
  • Teacher confidence gate is key

공통 Ways to turn a single outcome reward into turn- and token-level signals

None of the three methods trains a separate critic, but each needs different additional inputs.

Where things are heading: turn budgets, tool credit and rollout cost

Studies from November 2026 step back from 'performance goes up' to examine what improves and why. An analysis of the effects of multi-turn reinforcement learning paper evaluated with Pass@(K,T), which looks at sampling width K and interaction depth T together. It found that the benefit of trajectory-level reinforcement learning is clear only when the turn budget is sufficient, and shrinks sharply when the budget is tightened. Agent reinforcement learning also did not uniformly improve the quality of single-step decisions; its strengths were concentrated in error recovery and endurance on long tasks. The authors proposed BudgetMix RL, which varies the interaction budget during training, as an alternative, and reported stable performance across diverse execution environments.

TACA paper dug into the phenomenon of reasoning performance and tool-calling ability actually declining during training. It identifies three interlocking causes. When incorrect trajectories use more tool tokens per turn, the credit for tools overall becomes negative; token-mean aggregation gives more weight to longer incorrect trajectories; and retries that call a tool again after an execution error lengthen the trajectory, so even recovery attempts receive negative credit. TACA adds Advantage Projection, which shifts advantages minimally and uniformly within a prompt group so that the mean advantage of tool-call tokens does not go negative; Decision Contrast, which compares answers that used a tool with direct answers to the same question; and Supervised Replay, which retrains on the calls from previously correct trajectories.

System-side costs have also become a challenge. BunshinRL paper observed that agent rollouts interleave tool calls within persistent workspaces, so the memory of concurrently running environments grows and a single slow tool call can delay an entire batch. On top of BunshinFork, which clones an entire container workspace live, it added cross-environment memory sharing, tool-prefix caching and speculative tool execution, and reported reducing peak memory by up to 43%, total memory use by up to 51%, and end-to-end rollout latency by 23%. Methods that run more rollouts, such as branch comparison, are likely to benefit more from these system improvements.

Limitations and open issues: what to check in practice

The first thing to check is the evaluation setup. As the analysis of multi-turn reinforcement learning effects paper showed, gains on benchmarks with generous turn budgets can shrink considerably in real deployments with tight budgets. First align the maximum number of turns used in training and evaluation with the turn and time limits of the deployment environment. The same study discussed, at a preliminary level, that process supervision signals help but have limits, so it is better not to expect dense signals to solve this problem outright.

Second, confirm that your task has the ingredients a credit assignment method requires. Without a correct-answer string, TRACE paper-style value estimation is difficult; without task-specific skill documents, the teacher in privileged self-distillation becomes weak. As SDAR paper reported, simply combining reinforcement learning with OPSD can actually cause instability, so first consider a design that keeps the teacher signal as an auxiliary objective and controls its strength with a gate.

Third, a common mistake is aggregating the loss as a token mean without tracking how credit for tool calls evolves. As TACA paper showed, when long, incorrect trajectories dominate updates, the model can learn to avoid using tools. Rather than watching only accuracy on the training curve, log tool-call frequency and the mean advantage of tool tokens as well, so this degradation is caught early. For agents that converse with users, including user burden such as the number of questions in the reward or evaluation, as BAO paper did, helps avoid agents that 'only ask questions.'

Fourth, cost. In agent training that spins up a sandbox environment for every rollout, environment memory and slow tools tend to become bottlenecks more than model computation. BunshinRL paper's figures were measured on that particular system, so rather than carrying them over directly, first measure what is holding up batches in your own environment. Width-expanding parallel subagents in the style of WideSeek-R1 paper amplify the same cost problem.

Checklist for long-horizon agent RL

Before design

  • Do the turn budgets for training, evaluation and deployment match?
  • Is there a ground-truth answer string or skill document?
  • Is user burden included in the reward and evaluation?

During training

  • Is tool-call frequency declining?
  • Is the mean advantage of tool tokens negative?
  • Do long, incorrect trajectories dominate the updates?
  • Is the teacher signal gated as an auxiliary objective?

Cost

  • Are environment memory and slow tools the bottleneck?
  • Does cost grow with the number of parallel subagents?
Items to check before and during training.

Further study

If you want to pin down the problem definition first, start with the problem statements in TRACE paper and BranPO paper. Why good actions in failed rollouts get penalized and what non-monotonic correctness means are the starting points for the whole field. Then put the two papers' solutions side by side and consider what 'evaluating by value change' and 'branching and comparing' each trade off.

For the privileged self-distillation line, reading SDAR paper → ADRS paper → AgentOPSD paper in that order shows how the mechanism for adjusting 'how much to trust' the teacher signal is refined from a token gate to a reward-relevance gate to turn-level belief updates. If you are considering practical application, it is strongly recommended to read the analysis of multi-turn reinforcement learning effects paper together with TACA paper. They show under what conditions the performance gains reported by earlier studies can be trusted.

To broaden your view of agent behavior and structure, look in turn at BAO paper on user burden, WideSeek-R1 paper on widening with parallel subagents, and BunshinRL paper on training infrastructure cost.

Related terms

Reinforcement learning RLVR Distillation Tool use AI agent Multi-agent Sandbox Reward hacking

References

Foundational papers

Recent research

Written by AI from the listed references and reviewed. Please check the original sources for details.