Paper review · October 31, 2026
Youtu-Parsing-Omni: Parsing every modality with one encoder and one schema
Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities
A 5B parser that reads all modalities with a single shared encoder and unifies output into a single schema, achieving state-of-the-art performance in document, audio and video parsing.
Problem
For agents to find and cite evidence in documents, images, audio and video, they need structured records marked with location and time. But existing parsers use different models and output formats for each input type. Existing omni-modal models use separate vision and audio encoders, so the two modalities do not mix at the perception stage, and they also assign time axes separately.
Method
A shared Transformer encoder initialized from a pretrained language model (Youtu-Omni-Encoder) processes images, audio and video with the same weights and a single (t,h,w) positional encoding. In some fusion layers, video frames attend to audio from the same moment within a short time window. Outputs for 10 parsing tasks are unified into a JSON schema (OmniSchema) with spatial bboxes or time spans attached. After SFT and per-schema RLVR, OSAD is applied. OSAD is on-policy distillation in which the model trains its own rollouts toward token-level targets from a copy of itself (the teacher) that additionally sees the ground-truth parse, and the teacher is updated with the trained weights every round.
Results
- The authors reported that a single 5B checkpoint achieved the best performance, with an Overall score of 96.96 on OmniDocBench v1.6.
- It averaged 75.08 on OmniParsingBench, ahead of Logics-Parsing-Omni (72.26), ranking first among open-weight models, with gains concentrated in audio and video.
- Its overall chart parsing score of 95.02 was the highest among compared systems.
- In chemical structure recognition, its exact fingerprint match rate ([email protected]) of 54.2 beat DeepSeek-OCR 2 (49.6).
- Its printed sheet music transcription CER of 22.73 was on par with the specialized model LEGATO (23.30 as reported in the original paper).
Limitations
On OmniParsingBench it ranked first among open-weight models, not first overall including closed models. The LEGATO figure is cited from the original paper, so the comparison was not made under the same conditions. Audio and video timestamp resolution is 1 second. Structurally, adding a new modality requires only a stem and a projector, but the shared weights must be retrained. The provided text was cut off before the experimental details, so the ablation results could not be verified.
Why it matters
Once raw media is converted a single time into text records with location and time information, even text-only LLMs can read those records and trace evidence. It is notable in that a single model can run the perception layer for multimodal RAG and agents.
Key terms
- Omni-modal parsing
- The task of converting documents, images, audio and video into structured text records with spatial locations or time spans attached.
- On-policy Distillation
- A training approach in which the student targets the teacher's token-level distribution on outputs the student generated itself. It allows training under the same conditions as inference.
- OmniSchema
- A JSON schema that unifies the outputs of 10 parsing tasks. Each element carries a bbox or timestamp so its source can be traced.
Original
Authors: Yunhang Shen, Zhangquan Chen, Xiaotian Li, Zhongpeng Cai, Qing Zhou, Zekai Zhang, Kun Zhang, Zezhong Wu, et al. · Posted 2026-10-29 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.