ChronoReplay: One answer format is not enough for streaming spatial intelligence

Problem

When a spatial VLM is post-trained on a single answer format (e.g., choosing left/right), performance improves only in that format. It transfers poorly to numerical queries such as distance or temporal queries such as timing, or even drops. The authors call this 'interface lock-in'. In addition, numerical and temporal answers only have meaning relative to the history of past observations (prefix) that produced them, so splitting question-answer data into independent pairs makes the answers ambiguous.

Method

Starting from Qwen3-VL-8B, the model is post-trained in two stages. (1) Causal Replay construction: answers keep their original format, including options, units, precision and timestamps, and multi-round supervision is arranged in the order of the source video's timeline so that each answer is attached to a valid causal prefix. (2) Family-balanced learning: the losses of single-turn retention data (33,180 samples) and numerical/temporal Trajectory data (6,381 episodes) are normalized within each family and then mixed at a 1:2 ratio (α=0.5). No separate memory module or extra labels are used. At inference time, adaptive frame selection can be applied separately while the trained model stays frozen.

Results

• Under the same observation setting (Uniform-128), the OVO-S-Bench score rose from 48.98 for the base Qwen3-VL-8B to 53.89, numerically close to the published reference value for Qwen3-VL-235B-A22B (53.64).
• The authors reported gains over the base model of +13.41 points on OST-Bench, +7.44 points on UCS-Bench and +4.91 points on OVO-S-Bench, and the highest three-benchmark average among the supervision settings compared.
• Training on categorical data alone reached 58.03 on OVO-S, but OST fell from 50.52 to 47.48, showing interface lock-in.
• Time-ordered replay scored about 2.41 points higher on OST than a control that cyclically shifted the order of turn blocks, across two seed pairs. Family-balanced learning outperformed plain joint SFT and natural turn weighting.
• Adding adaptive frame selection raised OVO-S to 57.98 (80.5 frames on average), giving an inference-time gain with less visual input.

Limitations

Evaluation is limited to indoor scenes, so other environments are untested. The time-ordering effect was confirmed with only two seed pairs. The authors note that the single-family training controls also differ in data source and task scope, so the performance gap is hard to attribute to answer format alone. Qwen3-VL-8B is the only backbone model tested.

Why it matters

Robots and AR devices must answer questions about direction, distance and timing from the same video stream. This work shows that several query formats can be improved together without a new architecture, just by changing the format and order of the data and the weights across families. It is a useful reference when designing post-training data for spatial VLMs.

Key terms

Interface lock-in

The authors' name for the effect where post-training on one answer format concentrates gains in that format, with little transfer to other formats.

Causal prefix

The frames observed and the prior conversation history up to the moment a question is asked. In streaming evaluation, the model must answer using only this range.

Family-balanced learning

A training approach that normalizes the loss separately for each data family and then mixes them at a fixed ratio. It keeps families with many samples from dominating the model.

Original

• arXiv abstract
• PDF
• HTML full text
• Code
• Project
• Hugging Face

Authors: Yihao Hu, Yitong Lin, Pan Wang, Zhihao Wen, Hang Wang, Yiwen Zhao, Deyuan Liu, Yutong Li, et al. · Posted 2026-10-25 · License arXiv 비독점 배포 라이선스 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.