Paper review · October 15, 2026
VideoJev: A System One Model for Video Understanding and Interaction
VideoJev: System One Model for Video Understanding and Interaction
The authors replaced the generative answer decoding of a multimodal LLM with a candidate-scoring head and reused video computation across multiple queries. This improved temporal grounding accuracy while making video QA more than 100x faster.
Problem
Video multimodal LLMs generate timestamps or answers token by token even when the output is a short structured decision, such as temporal grounding or multiple-choice QA. This makes them slow. Agents query the same video repeatedly, and the extra cost incurred each time is also large.
Method
Qwen3.5-4B and Qwen3.6-35B-A3B (MoE) serve as backbones. The final MLP and output head are removed, the hidden representations of new tokens (<|option_end|>, <|decide|>) are projected, and candidates are scored by cosine similarity. For temporal grounding, frame-pair segments are fed in as timestamp text candidates, trained with BCE and soft IoU, and a single contiguous segment is decoded with Kadane's algorithm. Multiple-choice QA scores each option and trains with softmax and cross-entropy, shuffling option order to reduce position bias. It supports Cached inference, which computes the video prefix once and reuses it, and Joint inference, which handles multiple queries in a single forward pass. Training is split into stage 1, which aligns only the head, and stage 2, which also trains the backbone.
Results
- On average mIoU across 3 temporal grounding benchmarks, VideoJev-4B scored 65.3, 6.7 points higher than its backbone Qwen3.5-4B (58.6) and 3.3 points higher than TimeLens2-4B (62.0)
- VideoJev-35B-A3B reached an average mIoU of 67.7, 10.8 points higher than its backbone Qwen3.6-35B-A3B (56.9). Latency per query was 0.336 s, about one third of the backbone (1.054 s)
- On the average of 4 video QA benchmarks, VideoJev-4B scored 74.3, similar to Qwen3.5-4B in Thinking mode (74.9). Latency per question was 1.123 s, more than 100x faster than Thinking mode (144.9 s)
- Under the same data and settings, 1.3 points higher in average mIoU and 1.9 points higher in average QA than autoregressive fine-tuning
Limitations
QA covers only multiple choice, and the input is video frames only (no subtitles or audio). The QA training data is about 240,000 internally built questions, which makes reproduction difficult. Latency measures only model forward time and excludes preprocessing and data transfer. On some QA benchmarks it is slightly below Thinking mode.
Why it matters
When a video agent repeatedly checks progress or locates evidence, deciding directly from scores without generation can cut latency sharply. It is a practical interface for building real-time video-based workflows.
Key terms
- Temporal grounding
- The task of finding the time segment in a video where an event described in natural language occurs.
- mIoU(mean IoU)
- The average degree of overlap (intersection over union) between the predicted segment and the ground-truth segment. Higher means the location was found more accurately.
- Cached / Joint inference
- An approach that computes the beginning of the video once and reuses it for each later query (Cached), and an approach that concatenates multiple queries and scores them all in a single forward pass (Joint).
Original
Authors: Yuhan Zhu, Xiangyu Zeng, Qingyi Si, Chenxu Yang, Yifei Li, Luozheng Qin, Junhao Zhou, Dingyu Yao, et al. · Posted 2026-10-13 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.