Paper review · November 6, 2026
TurnWise: Improving Reinforcement Learning for Streaming Video with Turn-Level Credit Assignment
Turns Matter: Advancing Streaming Video RL via Turn-Level Credit
The method splits GRPO's trajectory-level advantage into turn-level advantages and averages per-turn losses with equal weight, so streaming video models learn better when to answer and when to wait.
Problem
A streaming video understanding model has to decide at each turn, while video is still coming in, whether to answer or keep watching. The standard GRPO approach assigns one advantage for the whole trajectory equally to every turn. As a result, a repeated answer given after an earlier success can receive a positive signal, while a useful follow-up answer given after an earlier failure can receive a negative one. In addition, with token-averaged loss, long responses get more weight than short waiting actions.
Method
(1) Silence-completed prefix scoring: keep only the actions up to a given turn, fill all later turns with "no response," and score the result with the same reward function. Subtracting this score from the full reward gives the reward earned from that turn onward (return-to-go), which is compared within the group to produce per-turn advantages. Because it reuses rollout evaluations that have already been computed, no critic model or extra rollouts are needed. The correction strength α is raised from 0 to 1 over the first 10% of training. (2) Turn-mean policy optimization: token losses are first averaged within each turn, then averaged across turns. Answer and wait decisions get equal weight regardless of response length. The reward is MMDuet2's PAUC-based reward, used unchanged, so that only credit assignment and loss aggregation are the variables being compared.
Results
- The authors report that on both Qwen backbones, the method achieved the highest overall scores on ProactiveVQA, OVO-Timing and StreamingBench, ahead of existing reinforcement learning methods such as MMDuet2 and ThinkStream.
- For reference, with Qwen2.5-VL-3B, the overall ProactiveVQA PAUC is 22.15 for the base model, 28.74 for SFT and 39.60 for MMDuet2. The TurnWise figures are cut off in the provided text and could not be confirmed.
- The authors report that, compared with using credit correction alone, adding turn-mean aggregation improved response timing and reduced repeated answers.
Limitations
Experiments fixed the reward function to MMDuet2's PAUC reward alone, so it has not been confirmed whether the effect holds under other reward designs. The experimental backbones are two Qwen-family models. The return-to-go calculation only works when there is an explicit waiting action ("set all later turns to no response") and a shared observation schedule, so it is hard to apply directly to tasks without that structure. Detailed TurnWise figures are cut off in the provided text and could not be confirmed.
Why it matters
It shows a cheap, critic-free way to fix a problem in multi-turn interaction: splitting the trajectory reward equally across all turns reinforces repeated answers. It is a useful reference for services where "when to speak" matters, such as real-time video assistants, and for other GRPO training with a turn structure.
Key terms
- GRPO(Group Relative Policy Optimization)
- A reinforcement learning method that samples several rollouts for the same input and normalizes each reward by the group mean and standard deviation to use as the advantage. It does not need a value function (critic) model.
- Credit assignment
- The problem of working out which of several actions the final reward should be attributed to. This paper does it at the turn level.
- Return-to-go
- The sum of rewards obtained from a given point to the end. It is used to exclude rewards received earlier and look only at the contribution of the current decision.
- Streaming video understanding
- A task in which the model keeps deciding whether and how to respond while frames are still arriving, rather than after watching the whole video.
Original
Authors: Shuai Dong, Kanghui Tian, Linghan Chen, Wenqi Shao, Hongrui Wu, Haowen Hou, Yongfu Zhu, Xuehui Wang, et al. · Posted 2026-11-04 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.