OmniCompass: A Comprehensive Benchmark for Jointly Evaluating Omni-modal Video Grounding and Understanding

Problem

Existing video understanding benchmarks cover only some evaluation axes. They look only at visual evidence or only at audio-visual evidence, specify targets by text alone, and lean toward either QA or grounding. As a result, it is hard to tell whether a model failed to find the target, failed to locate it, or failed to understand the content.

Method

The authors built two tracks. OmniCompass-VL (820 videos, 2,460 instances) contains tasks that can be solved with visual evidence alone, and OmniCompass-AV (240 self-collected movie, TV, variety show and news videos, 720 instances) contains tasks that also require the original audio. Each video is annotated with temporal grounding (when), spatial grounding (where) and QA (what). Every instance comes in a pair: one version specifies the target by text only, and the other by text plus a reference image. The authors newly defined CS (Cross-Specification Consistency Score), which looks at both the accuracy under the two settings and the agreement between their predictions. Four expert annotators spent about 1,800 hours labeling the data by hand and verified it over several rounds.

Results

• The authors reported that on the OmniCompass-VL overall score, Gemini-3.8-Flash (42.8%) was the highest among commercial models and Kimi-K3 (39.6%) was the highest among open-source models.
• The gap in average CS between the best commercial model and the best open-source model was 3.2% on the VL track and 32.8% on the AV track. Open-source models fell behind especially in audio-visual understanding.
• Only 11 of 36 models scored higher in the Text+Image setting, which adds a reference image, than their single-setting average.
• Even with high QA performance, temporal and spatial grounding remained weak, and predictions often disagreed when the same target was specified in different ways.
• In additional analysis, adding timestamps substantially improved performance across all tasks and tracks. On the AV track, adding audio consistently improved QA and temporal grounding performance.

Limitations

The AV track is small, with 240 videos and 720 instances. Labeling was done by four annotators and one lead expert. Inputs are uniformly sampled frames, so results may depend on input settings such as the number of frames. The provided text is cut off partway through the construction process, so the detailed experimental settings and the limitations stated by the authors could not be checked.

Why it matters

When deploying a video MLLM in a service, this makes it possible to measure separately the failure where 'the answer sounds plausible but the model cannot pinpoint the segment or location.' It also shows whether predictions stay consistent across ways of specifying the target, which helps with model selection for practical tasks such as video search and highlight extraction.

Key terms

Temporal/Spatial Grounding

A task of finding the time segment (temporal) in which the target referred to by a question appears in a video, or its location box on screen (spatial).

Omni-modal model

A multimodal model that processes not only video frames but also audio and text together in a single model.

CS(Cross-Specification Consistency Score)

A metric that checks whether the two predictions, one made when the target is specified by text only and one made when it is specified with an accompanying image, are both correct and agree with each other. The score drops if either one is wrong or if they disagree.

tIoU

A ratio (intersection/union) showing how much a predicted time segment overlaps with the ground-truth segment.

Original

• arXiv abstract
• PDF
• HTML full text
• Code
• Project
• Hugging Face

Authors: Xuyang Liu, Shihao Wang, Xinyu Wei, Minghan Li, Zeqing Wang, Zuhao Yang, Zichen Wen, Lei Zhang · Posted 2026-10-13 · License CC BY 4.0 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.