Paper review · October 9, 2026
Which Rollout Taught That Behavior? BehaviorTrace, Testing the Limits of Training Data Attribution in Online Reinforcement Learning
Which Rollout Taught It That? BehaviorTrace and the Limits of Training-Data Attribution in Online RL
In controlled GRPO training experiments with a planted behavior of known origin, the authors show that much of the signal from gradient-based training data attribution comes from confounding factors.
Problem
Reinforcement learning fine-tuning can teach unintended behaviors. Gradient-based attribution techniques such as influence functions, TracIn and TRAK are used to find the training Rollouts that produced such behavior. However, these techniques were mostly validated in static, supervised learning settings, and whether they actually pinpoint the causes of behavior in online reinforcement learning has rarely been tested.
Method
Qwen2.5-1.5B-Instruct is trained with GRPO for 1,000 steps after a short SFT warm-up. A behavior with a known cause is planted by giving a bonus reward, only early in training, for outputting 'QZXBT' in response to a 'frobnitz' prompt. Full gradients are compressed to 65,536 dimensions with CountSketch so that behavior-related parameters are not lost, and existing estimators are evaluated, including GAS (renormalized TracInCP), a TRAK-style estimator and TracInCP. Controls include norm-only, which looks only at gradient magnitude without a behavior target, a fluency (mean token log-probability) baseline, and variation across 3 seeds and multiple generation Draws. The authors propose no new estimator but release an evaluation harness and checklist.
Results
- In step-level precision, the norm-only control with no behavior target scored 0.422/0.396/0.385 (seeds 0/1/2), similar to the best targeted estimator (GAS 0.404/0.383/0.421) and higher on 2 seeds. The authors report that these results are 4.2–4.5 times chance level.
- When narrowed to the window in which learning occurred, all methods, including random, were within about 0.03 of each seed's chance level. The achievable headroom at the step level was also only about 0.11. The authors interpret this to mean that step-level attribution finds only 'which step was learned', not 'which Rollout'.
- On seed 2's preregistered Draw (882 pairs), within-group AUC was 0.724 for the fluency baseline, 0.721 for norm-only and 0.669 for GAS, in that order. At the saturated checkpoint, fluency predicted as well as or better than all gradient methods.
- After controlling for fluency, GAS's directional AUC varied by seed at 0.486/0.624/0.629 (seeds 0/1/2). A newly generated Draw from the seed 1 checkpoint gave 0.526, reversing the verdict, so no conclusion could be drawn from a single run.
- At the token level, the cosine between targets built from real response contexts and the trigger-token gradient was positive in all three seeds (+0.114 to +0.295). In contrast, artificially constructed continuous targets were near zero or negative (−0.111 in seed 2).
Limitations
The study uses a single model, Qwen2.5-1.5B, three seeds, and a single implanted behavior: one single-token trigger. The behavior did not arise from reinforcement learning alone; reinforcement learning amplified a link first implanted through an SFT warm-up. The causal check of retraining without the correct rollouts was run not in the main experiment but in a similar setting where contamination was applied throughout training. The rollout-level analysis used samples newly generated from saved checkpoints, not the original training rollouts. Only seed 2 had 100 or more fluency-matched pairs.
Why it matters
For practitioners trying to trace the training data behind unwanted behaviors that emerge during reinforcement learning, the paper warns that the 'plausible' results produced by existing attribution methods may reflect confounders such as gradient magnitude or fluency. It provides a validation checklist, including gradient-magnitude controls, fluency controls, and repetition across multiple seeds and draws, along with a public harness. It also points to token-level attribution as a more promising direction.
Key terms
- Training-Data Attribution
- A set of methods that estimate which training samples caused a particular model output or behavior. TracIn and TRAK, which compare gradient similarity between training samples and a target behavior, are well-known examples.
- GRPO
- A reinforcement learning method that generates several responses (a group) for the same prompt and updates the policy using an advantage normalized by the mean and standard deviation of rewards within the group. If rewards within a group do not vary, the group contributes nothing to the update.
- Norm-only control
- A baseline that ranks samples by gradient magnitude alone, without a behavior target. If this control performs about as well as a target-based estimator, the estimator's performance likely comes from gradient magnitude rather than the behavior direction.
- CountSketch
- A compression technique that uses hashing to assign each coordinate of a high-dimensional vector to a bucket and a sign, then sums them. Inner products are approximately preserved, and because every parameter is included, signals present in only some parameters are not missed.
Original
Authors: Amit Nautiyal · Posted 2026-10-07 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.