Paper review · October 31, 2026
A-RSI: Adaptive task pool refresh for multimodal self-improvement without external data
A-RSI: Adaptive Task Refresh for Zero-Data Multimodal Self-Improvement
By deciding when to refresh the task pool from reward variance and accuracy in training logs, and training on program-computed answers, it raised the average across 11 benchmarks from 57.77 to 61.05.
Problem
Research on recursive self-improvement has mostly focused on what to generate next, with less attention to when the training data currently in use stops being useful. Also, when answers are produced by self-consistency or majority voting, the model's own errors can become labels for the next round of training.
Method
Before training, a library of fixed program templates written by the model generates images, questions and answers together from the same settings in the domains of geometry, charts, tables and counting. The solver is trained with GRPO, receiving correctness, format and termination rewards. At the end of each training window, a controller checks whether the number of prompts with remaining within-group reward differences (U_w) has fallen below a set fraction or whether training accuracy has risen enough, and if so, swaps the whole pool for a prepared, unused task pool. Model weights carry over, and validation and test scores are not consulted.
Results
- In experiments based on Qwen3-VL-8B, the average across 11 benchmarks rose from 57.77 (initialization checkpoint) to 61.05, using only 800 new tasks.
- Refreshing the pool every window used 3,200 tasks and reached 60.64, lower than A-RSI.
- At the same sixth window, A-RSI scored 60.40, while continuing to train on the existing pool scored 58.35.
- On the MiMo model, the score also rose from 59.08 for the public base model to 60.71.
- The authors reported that refresh rules accounting for difficulty and capability were no better than random sampling with fixed domain proportions.
Limitations
The authors state that "saturation" is only an operational rule based on training logs, not evidence that the pool has been mastered. Reward variance can also disappear when all responses are wrong or when responses collapse to one side. The templates and sampling distribution are fixed, and answer correctness depends on the template implementations being correct. The comparison of selection strategies is also limited to this generator, model and rule set. Part of the improvement may come from a higher rate of finishing answers within the output length limit, which the authors analyzed separately.
Why it matters
It shows a practical rule for reducing data costs when continuing reinforcement learning without human-labeled data, using only training signals to decide when to add new data.
Key terms
- GRPO
- A reinforcement learning method that compares the rewards of multiple responses sampled from the same prompt within a group to compute relative advantages. If all responses receive the same reward, the learning signal is zero.
- Task pool refresh
- Replacing the entire set of problems currently being trained on with a new, unused set of problems. Model weights carry over unchanged.
- Program-computed answers
- Because the program that created a problem computes the answer directly from the same internal settings, answers can be checked without model voting.
Original
Authors: Yu Liu, Youxing Li, Yuedong Tan, Zhiwei Yang, Zihan Su, Hanqi Feng, Shaojie Zhang, Libin Liao, et al. · Posted 2026-10-28 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.