Paper review · November 21, 2026
WeMM-Video: An open-source family of omni-modal video understanding models
WeMM-Video: Pushing the Frontiers of Open-Source Omni-Modal Video Understanding
Pretraining on about 1.2T tokens followed by 3-stage post-training produced omni-modal models from 2B to 35B-A3B, achieving top-tier performance among open-source models in audio-video understanding.
Problem
Understanding video with accompanying audio requires reasoning jointly about fine-grained temporal changes between frames and the correspondence between sounds and scenes. Existing open-source models missed key audio-visual details, failed to match sounds to the corresponding scenes or produced cross-modal hallucinations.
Method
The models use Qwen3.5 (2B/4B/9B) and Qwen3.6-35B-A3B as backbones, with a Whisper-based audio encoder and a window-level Q-Former (1 audio token per 0.1 seconds) attached. Video tokens and audio tokens are interleaved in temporal order, with text timestamps added. Pretraining has 3 stages: a stage that trains only the audio alignment module, an omni-modal knowledge injection stage that trains everything except the visual encoder, and a context extension stage. Post-training also has 3 stages: multi-Teacher OPD (On-policy Distillation) to improve general capabilities, MrDPO with Hard Negatives made of repetitive outputs to stabilize generation, and finally specialized reinforcement learning with rule- and rubric-based rewards. Both an instant-answer mode and a thinking mode are supported.
Results
- 35B-A3B (Thinking) outperformed Seed2.0-Mini-0428 on 16 of 17 general audio-video understanding benchmarks, and Qwen3.5-Omni-Flash on 16 of them.
- On Video-MME (with audio), 35B-A3B (Thinking) scored 86.4, higher than Seed2.0-Mini (83.6) and Qwen3.5-Omni-Flash (79.3) but lower than Gemini-3.0-Flash (88.6).
- On the same benchmark, 9B (Thinking) scored 85.1, ahead of Qwen3-Omni-30B-A3B-Instruct (77.3), and the authors reported that 9B surpassed this 30B-class model on most general audio-video tasks.
- Comparing the instant-answer and thinking modes of 35B-A3B, the thinking mode scored higher on most tasks (e.g., SciVideoBench 57.8→69.4).
Limitations
A gap with Gemini-3.0-Flash remains, and the models trail it on many benchmarks. The thinking mode is not always better: on FutureOmni (58.4→56.2) and JointAVBench (67.4→65.9) it scored lower than the instant-answer mode. Some figures for comparison models are the authors' own evaluations rather than official results. Captioning and interaction evaluations rely on LLM judges such as GPT-family models. The detailed composition of the training data and whether it will be released could not be confirmed from the excerpt.
Why it matters
The authors released open models that jointly understand audio and video, starting from small sizes (2B–9B), together with evaluation and reproduction pipelines. They are a practical starting point for teams that want to build production video AI themselves, such as for live commerce or short-form video analysis.
Key terms
- Omni-modal
- An approach in which a single model takes text, images, audio and video as input together and processes them.
- OPD(On-policy Distillation)
- A Distillation method in which the Teacher model provides supervision signals on outputs the Student model generated itself. This paper used multiple Teachers split by modality.
- MrDPO
- A preference optimization technique that repeats DPO over multiple rounds. Here it used repetitive and endless generation cases as Hard Negatives to reduce output repetition.
- Hybrid-thinking
- A setup that lets a single model switch between a fast instant-answer mode and a thinking mode that reasons step by step.
Original
Authors: Changli Tang, Jie Yang, Tianyi Wang, Lei Li, Daoze Zhang, Zijie Xin, Zhan Liu, Li Li, et al. · Posted 2026-11-19 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.