Paper review · November 14, 2026
Duo: A Reciprocal World Model for Real-Time Two-Way Conversation
Duo: A Reciprocal World Model for Full-Duplex Interaction
By tying a language model that decides intent and a Diffusion model that generates speech and video onto a single timeline, the authors built real-time two-way video conversation with the user, including interruptions.
Problem
Existing world models focused on simulating the physical world. Users could control the world only with low-degree-of-freedom inputs such as keyboard, mouse and text. There was no intelligence inside the world to perceive and respond to the user in real time.
Method
The authors proposed the Reciprocal World Model (RWM) and built Duo v1 as its implementation. The Intent Module is a multimodal language model that autoregressively outputs <speak>+text or <wait> for every 17-frame (about 0.71-second) chunk. The Realization Module is a causal audio-video Diffusion Transformer initialized from MiniMax-H3 that generates speech and video together, conditioned on Intent's hidden state. All inputs are aligned to a single timeline, and per-module attention masks block access to future information. Finally, Dual rollout Distillation is applied. DMD is performed conditioned on the text generated by Intent and the past audio and video generated by Realization itself, yielding a 4-step streaming generator.
Results
- They reported implementing full-duplex interaction that keeps perceiving the user's speech, video and text while responding chunk by chunk (about 0.71 seconds).
- Most of Intent's training data consists of chunks with no new utterance. The authors explained that per-action reweighting prevented a collapse in which the model always outputs only <wait>.
- They reported that Dual rollout Distillation produced a 4-step streaming generator that narrows the gap between training and actual inference conditions.
- Beyond conversation, they demonstrated scenarios such as showing real objects, changing virtual scenes and audio-video games. The model has been released.
Limitations
This is a technical report. The provided text contains no quantitative evaluation of latency, generation quality or conversational appropriateness, and no baseline comparisons. The authors themselves present v1 as a first step. A large share of the training data is scripted, synthesized video calls.
Why it matters
It shows an architecture that combines interfaces that see, hear and speak in real time, such as AI avatars or virtual agents, into a single system. The design in which the language model directly decides 'speak or wait' is a useful reference for handling natural interruptions and conversational timing.
Key terms
- Full-duplex
- A communication mode in which both sides can speak and listen at the same time. One side can interrupt or react while the other is still speaking.
- Intent / Realization
- Intent is the cognitive module that decides when to say what. Realization is the generation module that expresses that intent as speech and video.
- DMD(Distribution Matching Distillation)
- A technique that distills a Diffusion model that needs many steps into a Student model that generates in fewer steps. It trains by matching output distributions.
Original
Authors: Hengyuan Cao, Biao Gong, Guoshan Lu, Qiyuan Zhang, Lin Liu, Yiding Tian, Yilei Jiang, Yuxin Ma, et al. · Posted 2026-11-11 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.