Speech AI and real-time voice agents: how to think while listening

Why it was needed: architectures that can't keep up with the rhythm of human conversation

When people talk to each other, they prepare their reply before the other person has finished. They insert backchannels such as "yeah" or "mm" along the way, and cut in when needed. Voice agents (AI agents that hold spoken conversations while handling tasks such as bookings and lookups) feel awkward mostly because they cannot follow this rhythm.

The most widely used architecture is the cascade (a chain of stages). Speech recognition (ASR) transcribes the speech, an LLM writes the answer, and TTS (speech synthesis) reads the answer aloud. Each component can be chosen separately, which makes it easy to build. But waiting time accumulates at every stage. The endpoint anticipation study paper points out that a cascade creates a bottleneck because it detects 'whether the speech has ended' only after it has ended. The LLM starts work only after the user stops talking, so a silence appears in between.

Information lost the moment speech is converted to text is another problem. WavReward paper notes that spoken conversation carries a great deal of non-textual information that text-based language models struggle to measure. Tone of voice, emotion, moments when two people's speech overlaps, and interruptions do not all fit into a single line of transcript.

Core idea: from walkie-talkie to phone

Think of a walkie-talkie. One side has to say "over" before it is the other side's turn to talk. This mode, in which only one side speaks at a time, is called half-duplex. A phone is different. Both people can speak and listen at the same time, and one can respond with "oh, right" while the other is still talking. This is full-duplex. The Japanese full-duplex study paper describes full-duplex dialogue systems as systems that model the simultaneous, bidirectional nature of human conversation, such as overlapping speech and backchannels.

There are two main ways to implement this idea. One keeps the cascade but makes each stage move 'ahead of time'. The other uses an end-to-end speech language model, which converts speech into tokens (the small units a model works with) and predicts the next sound directly, the way an LLM predicts the next word. NTPP paper focuses on dual-channel data, in which the two speakers' voices are recorded separately on different channels. Such data, it argues, preserves the structure and flow of the conversation as they are.

The trade-off between the two approaches is clear. In a cascade, the transcribed text remains as an intermediate result, which makes inspection, logging and tool integration easy. In exchange, waiting occurs and nonverbal information is lost. End-to-end full-duplex gains a natural rhythm, but it is hard to look inside and may be weak at complex reasoning and tool use.

How it works: listening and speaking at once, and moving ahead of time

An end-to-end full-duplex model runs roughly as follows. First, it converts incoming speech into speech tokens in short chunks. Second, at every moment the model looks at both the 'user channel' and its 'own channel'. NTPP paper proposed pairing the tokens of the two channels and predicting the next pair (Next-Token-Pair Prediction). As a result, even a decoder-only architecture could learn speaker-agnostic dual-channel dialogue, and the study reported much lower inference latency than previous methods. Third, the model decides for itself whether to speak at this moment, wait, or just backchannel. As the study on aligning with user conversations paper points out, real-time speech is full of interruptions and fillers ("um", "uh") and has no explicit boundaries between speaker turns. So this decision itself has to be learned.

On the side of making cascades faster, the key is 'start before it ends'. Endpoint anticipation (Endpoint Anticipation, paper) looks at the speech and predicts in advance when the turn will end, instead of detecting after the fact whether speech has ended. The model predicted endpoints up to 2.56 seconds ahead. This makes it possible to run the LLM and TTS through speculative execution (computing ahead of time while knowing the result may be wrong) using context that has not been fully heard yet. When attached to the Unmute framework, average latency dropped by 505 ms and speculative computation rose by 28.4%.

These numbers show the basic trade-off in this field. Reducing latency means accepting computation that gets thrown away when a prediction misses. For this reason, the study separately proposed a metric that weighs the latency actually saved against redundant computation.

How the field developed: learning, refining and evaluating conversational structure

On learning conversational structure, NTPP paper systematically applied dual-channel speech data to modern LLM methods. The approach improved turn-taking prediction, response coherence and naturalness together. The Japanese full-duplex study paper released a Japanese model based on Moshi, an English full-duplex model. It used a two-stage approach: pretraining on large-scale Japanese spoken dialogue, then fine-tuning on high-quality stereo dialogue data. Synthetic data generated by a TTS that synthesizes multiple speakers at once was also added to training. This showed that synthetic data can fill gaps even for languages with little data.

On refining models, the study on aligning with user conversations paper carried preference learning, previously text-centered, over to real-time speech. It built more than 150,000 preference pairs with AI feedback from real multi-turn spoken conversations, capturing preferences not only about what is said but also about temporal context such as when to interrupt. Aligning a full-duplex speech-to-speech model offline with this data made its responses more factual, safer and better suited to context. TiCo paper addressed the ability to follow time instructions such as "answer in about 15 seconds". It inserted markers that tell the model 'how many seconds it has spoken so far' during generation (Spoken Time Markers) and post-trained with RLVR (reinforcement learning with verifiable rewards). As a result, length error fell by 2.7x compared with the base model and by 1.6x compared with the strongest baseline.

On evaluation, WavReward paper built a reward model that listens to speech input directly and scores a dialogue model's intelligence (IQ) and emotional responsiveness (EQ) together. It was trained on the preference dataset ChatReward-30K, and objective accuracy rose from Qwen2.5-Omni's 53.4% to 91.5%. In this way, research in 2025–2026 broadened from building 'models that speak naturally' to deciding what to refine those models with and what to measure them by.

Where things are heading: agents that work while listening

The focus of recent research is shifting from models that speak well to agents that also get work done while listening. The early tool calling study paper changed the key question from 'which tool to call' to 'when has enough been heard'. It built 26,694 synthetic spoken conversations (786 hours, 48 domains) labeled with the point at which the tool and arguments needed for a request first become clear (the earliest actionable point), and trained a multimodal model on this data. Tool-state macro-F1 was only 0.48 and 0.47. But counting only correct calls, the model acted on average 1.8–2.0 seconds (in-domain) and 1.1 seconds (external benchmark) earlier than a cascade that waits for the turn to end.

Heartbeat Thinking paper pointed out a weakness of existing speculative approaches: if generation is cut off every time new speech arrives, tool calls get cut off before their arguments are filled in. In this study, new input launches a new reasoning pass, while reasoning already in progress and tools already running are allowed to finish. Periodically run reasoning reads the latest partial transcript together with a shared record that collects the reasoning and tool results completed so far. So preparatory work continues across speech updates. On FDBench v3, tool coverage rose by 16 percentage points and final response latency fell by 36.4%. However, the effect on final task quality varied by benchmark.

DuplexAgent-RSI paper set out the collaboration pattern the industry is converging on. The full-duplex model stays on the real-time channel as the conversational front end, and hands off search, reasoning and coding asynchronously to other agents. A harness that connects the two (a layer that wraps the model and coordinates the workflow) handles accepting tasks, reporting progress, cancelling, replacing and delivering results. The study split the harness into six modules that can be modified. It then built a closed loop in which an LLM revises the modules based on failure logs produced by a simulator.

Limitations and open issues: what to weigh when choosing

First, moving ahead of time has a cost. Endpoint anticipation paper cut latency by 505 ms but increased speculative computation by 28.4%. Heartbeat Thinking paper also reduced latency, but its effect on final quality varied by benchmark. Tasks whose results can be thrown away, such as lookups, are worth doing ahead of time. By contrast, for hard-to-reverse tasks such as payments or booking cancellations, the sensible design is to block them separately so they do not run before the request is confirmed. The macro-F1 of around 0.5 for early tool calling paper also shows that 'acting while listening' is still at an early stage.

Second, evaluation is hard. On TiCo-Bench, built by TiCo paper, existing open-source and commercial models often failed to meet explicit time constraints. WavReward paper was created because text-based graders cannot measure the nonverbal information in speech. Judging a system 'fast' from average latency alone is a common mistake. Whether it properly yields to interruptions, and whether it stops talking unnecessarily at backchannels, need to be tested separately.

Third, there is heavy reliance on synthetic data. Both the Japanese model paper and early tool calling paper used synthetic spoken conversations. Synthetic conversations may differ from the rough conditions of real calls, such as noise, regional accents and several people talking at once. Results need to be checked again in real production environments.

The practical criteria for choosing are as follows. If there are many tool calls and logging and auditing matter, a reasonable starting point is a cascade, which keeps text as an intermediate result, with endpoint anticipation and speculative execution added. If natural conversational rhythm is central to the product, use a full-duplex model. If both are needed, consider a setup like DuplexAgent-RSI paper that separates the conversation role from the task role and coordinates between them with a harness.

Further reading

If you are new to the topic, start with endpoint anticipation paper. It gives the most concrete picture of where the bottleneck in a cascade comes from and how speculative execution trades computation for latency. Then read NTPP paper and the Japanese full-duplex study paper to get a sense of how end-to-end full-duplex models learn conversational rhythm from dual-channel data.

If you want to know how models are refined and evaluated, read the study on aligning with user conversations paper, WavReward paper and TiCo paper, in that order. They cover preference learning, speech-specific reward models and time control, respectively.

To move on to agents, read early tool calling paper and Heartbeat Thinking paper together and compare 'when to act' with 'how to continue work already in progress'. Finally, read DuplexAgent-RSI paper to see how these components fit together in a real system.