Confucius4-R2T2: A Streaming Speech Recognition Model That Never Revises Its Output

Problem

Streaming speech recognition output is increasingly consumed directly by machines such as translators and voice agents rather than by people. If an emitted token is later revised, work based on that token may be wasted or an agent may act incorrectly. Existing methods only reduce revision frequency through inference-time commit rules or indirect training, and give no guarantee of zero revisions. There is also residual interference, where newly appended trailing audio destabilizes earlier tokens when audio is fed in incrementally from the start.

Method

Audio is fed incrementally into a full-context model with a frozen encoder. Among the outputs produced up to each point, the longest one that is entirely a prefix of the final full-context output is used as the LSP label. The authors prove that this label never shrinks over time (monotonic), equals the full output at the end, and is the longest among outputs the model actually produced. The decoder is trained to emit a Wait token when there is not yet enough audio, separating when to emit from what to emit. The AuT audio encoder and projector are frozen, and two-stage post-training was done on 1,500 hours of Chinese and English audio. Noise and reverberation augmentation and language tag flipping were also applied. No extra layers or attention masks are used, and chunk size can be adjusted from 80 ms to 2 seconds.

Results

• The authors report that R2T2 guarantees output stability (it never revises already-emitted output) while bringing streaming recognition accuracy close to the base model's full-context quality.
• They claim it produces stable text with low output latency and is state-of-the-art in accuracy and latency even compared with commercial models.
• They report improvements even in languages not used in post-training.
• They prove theoretical properties (monotonicity, consistency with the final output, maximality) comparing LSP with two existing definitions (all-future and final-anchored).

Limitations

The provided text is cut off after the training data section, so specific figures such as WER and latency, and comparisons with baselines, could not be checked. An LSP label sets the credit at a given point to 0 if even a single token differs, which biases the model toward emitting output late (the authors acknowledge this). Because the model uses a bidirectional encoder, the encoder must be rerun on the growing audio for every query, which adds compute cost. The post-training data is private Chinese and English data.

Why it matters

In systems that act on recognition results right away, such as real-time translation, voice agents and live captioning, output that does not flip is as important as accuracy. This work shows a practical path to turning an existing full-context ASR model into a stable streaming model without changing its architecture. The model weights and code have also been released.

Key terms

Streaming ASR

Speech recognition that keeps producing text while audio is still coming in. The key is balancing accuracy, latency and output stability.

Stable prefix

The property that output at an earlier point always matches the beginning of later output, so text that has been emitted never changes.

LSP(Longest Stable Prefix)

The longest of the outputs the model has produced up to that point that is, in its entirety, a prefix of the final output. This paper uses it as the label for streaming training.

Wait token

A waiting signal the model emits when there is not yet enough audio to commit to the next token. It is part of the same vocabulary as ordinary text tokens.

Original

• arXiv abstract
• PDF
• HTML full text
• Code
• Project
• Hugging Face

Authors: Daniel Huang, Qi He, Shaoxiong Duan, Yingjie Li, Qiao Cheng, Yitao Duan · Posted 2026-11-04 · License arXiv 비독점 배포 라이선스 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.