Reframe: Observation-aligned multi-teacher on-policy distillation for agentic visual reasoning

Problem

Visual agents can call tools such as image cropping or frame selection yet not use the results in their reasoning, a problem called "observation neglect." In addition, MOPD (Multi-teacher On-policy Distillation) supervises the student's rollouts as they are, so when a tool call is wrong, distillation happens on top of useless observations. Existing SFT→RL approaches that handle images and video together have lagged behind single-modality models.

Method

Built on Qwen3.5-9B, the method uses two tools, CropImage and SelectFrames, and the 100,000-example Reframe-100K dataset. In stage 1, GD-GRPO, spatial and temporal expert models are trained separately. The model receives an observation reward based on IoU with the evidence region, and after the first tool call the original visual input is removed with 50% probability to make the model depend on the tool results. Advantages are normalized within subgroups formed by whether the input was removed. In stage 2, OG-MOPD, if the student's tool call is invalid or its IoU falls below a threshold, it is replaced mid-rollout with an action based on ground-truth annotations. Distillation then continues from the corrected observation, with an NLL loss applied to injected tokens and the MOPD loss applied to student-generated tokens. Neither stage requires an SFT cold start.

Results

• The authors report that across 14 benchmarks, including 7 image and 7 video benchmarks, it outperformed Mix-RL and standard MOPD and, on average, also surpassed the single-modality expert models.
• On V* it scored 91.62, higher than the base model Qwen3.5-9B (83.77) and standard MOPD (90.05), and on VisualProbe it scored 57.48, higher than MOPD (52.62). However, on HR-Bench-4K it scored 84.75, slightly below Image-RL (85.25).
• Observation reliance measured on V* was 75.53, far higher than DeepEyesV2 (41.93) and Thyme (34.20). Its tool-use gain across the 7 image benchmarks was +5.23, the largest among the compared models (Thyme +1.47, DeepEyesV2 −2.45).
• In ablations, the authors report that GD-GRPO increased observation reliance and tool gain compared with outcome-based GRPO.

Limitations

The tools are limited to CropImage and SelectFrames, and only one base model, Qwen3.5-9B, is used. OG-MOPD and the observation reward depend on spatial and temporal evidence annotations, so they are hard to apply directly to tasks without such annotations. Some of the video benchmark results are cut off in the provided text.

Why it matters

It shows that evaluating tool-using agents requires checking not only whether they called a tool but whether they actually used the result. It also offers a new design direction: intervening in the rollout itself during distillation to ensure observation quality.

Key terms

MOPD(Multi-teacher On-policy Distillation)

A post-training method in which rollouts generated by the student itself are sent to domain-specific teacher models to receive dense, token-level feedback.

Observation neglect

A phenomenon in which a model calls a tool and obtains an intermediate observation, yet its subsequent reasoning is barely affected by that observation.

GRPO

A reinforcement learning algorithm that samples multiple rollouts from the same input and computes advantages from relative rewards within the group. It does not need a separate value model.

IoU(Intersection over Union)

A metric of how much a predicted region overlaps with the ground-truth region. Here it is used to judge whether a tool correctly captured the evidence.

Original

• arXiv abstract
• PDF
• HTML full text
• Code
• Hugging Face

Authors: Haozhi Yuan, Handong Li, Yifan Zhang, Qianshan Wei · Posted 2026-11-09 · License arXiv 비독점 배포 라이선스 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.