← Key papers

Paper review · November 21, 2026

WeMM-Video: An open-source family of omni-modal video understanding models

WeMM-Video: Pushing the Frontiers of Open-Source Omni-Modal Video Understanding

MultimodalSpeech & audioLanguage models Intermediate Upvotes 39 cs.CV Tencent

Pretraining on about 1.2T tokens followed by 3-stage post-training produced omni-modal models from 2B to 35B-A3B, achieving top-tier performance among open-source models in audio-video understanding.

Problem

Understanding video with accompanying audio requires reasoning jointly about fine-grained temporal changes between frames and the correspondence between sounds and scenes. Existing open-source models missed key audio-visual details, failed to match sounds to the corresponding scenes or produced cross-modal hallucinations.

Method

The models use Qwen3.5 (2B/4B/9B) and Qwen3.6-35B-A3B as backbones, with a Whisper-based audio encoder and a window-level Q-Former (1 audio token per 0.1 seconds) attached. Video tokens and audio tokens are interleaved in temporal order, with text timestamps added. Pretraining has 3 stages: a stage that trains only the audio alignment module, an omni-modal knowledge injection stage that trains everything except the visual encoder, and a context extension stage. Post-training also has 3 stages: multi-Teacher OPD (On-policy Distillation) to improve general capabilities, MrDPO with Hard Negatives made of repetitive outputs to stabilize generation, and finally specialized reinforcement learning with rule- and rubric-based rewards. Both an instant-answer mode and a thinking mode are supported.

Results

Limitations

A gap with Gemini-3.0-Flash remains, and the models trail it on many benchmarks. The thinking mode is not always better: on FutureOmni (58.4→56.2) and JointAVBench (67.4→65.9) it scored lower than the instant-answer mode. Some figures for comparison models are the authors' own evaluations rather than official results. Captioning and interaction evaluations rely on LLM judges such as GPT-family models. The detailed composition of the training data and whether it will be released could not be confirmed from the excerpt.

Why it matters

The authors released open models that jointly understand audio and video, starting from small sizes (2B–9B), together with evaluation and reproduction pipelines. They are a practical starting point for teams that want to build production video AI themselves, such as for live commerce or short-form video analysis.

Key terms

Omni-modal
An approach in which a single model takes text, images, audio and video as input together and processes them.
OPD(On-policy Distillation)
A Distillation method in which the Teacher model provides supervision signals on outputs the Student model generated itself. This paper used multiple Teachers split by modality.
MrDPO
A preference optimization technique that repeats DPO over multiple rounds. Here it used repetitive and endless generation cases as Hard Negatives to reduce output repetition.
Hybrid-thinking
A setup that lets a single model switch between a fast instant-answer mode and a thinking mode that reasons step by step.

Original

Authors: Changli Tang, Jie Yang, Tianyi Wang, Lei Li, Daoze Zhang, Zijie Xin, Zhan Liu, Li Li, et al. · Posted 2026-11-19 · License arXiv 비독점 배포 라이선스 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.