Efficient streaming audio-visual target speaker extraction for real acoustic environments (Falcon) and the RealSSA benchmark

Problem

Real-time audio-visual target speaker extraction (AVTSE) is needed in latency-sensitive settings such as video conferencing and hearing aids. However, existing benchmarks fix the number of speakers at two, do not cover near-field and far-field interference, reverberation or transient noise, and have no evaluation set of real recordings. High-performing causal models are too heavy to run on edge devices. Lightweight models call a small separator repeatedly, and the added computation cancels out the savings.

Method

(1) RealSSA benchmark: RealSSA-Sim places 1–5 near-field interfering speakers in four scene types: small meeting rooms, medium indoor spaces, large public spaces and outdoors. Scenes with far-field interference add 10–15 far-field speakers. Reverberation (RT60), ambient noise and transient noise are sampled together. The dataset totals about 153 hours. RealSSA-Real is evaluation-only real data recorded with a single mobile device. (2) Falcon: after fusing lip video features with audio features, an asymmetric U-Net separator passes through the encoder–decoder only once, splitting frequency-axis, time-axis and joint time-frequency modeling across resolutions. (3) A speaker contrastive loss, which contrasts the clean target speech with each near-field interfering speaker in the separator's STFT encoder space, is used only during training.

Results

• The authors report that Falcon matched or exceeded the separation quality of the strongest causal baselines (AV-TFGridNet, AV-Mossformer2, SwiftNet and others) on RealSSA-Sim, LRS2-2Mix, LRS3-2Mix and VoxCeleb2-2Mix.
• Compared with causal AV-TFGridNet: 91.9% fewer parameters, 20.2× less computation and 11.1× lower GPU inference latency.
• On the real-recording set RealSSA-Real, the authors report competitive perceptual quality (DNSMOS, NISQA), WER and speaker similarity.
• The authors report that the speaker contrastive loss reduces near-field interference leakage at no inference cost and also works with other AVTSE backbones.

Limitations

RealSSA-Sim is a single-channel 16 kHz simulation. Target speakers come from LRS3 and interfering speakers from LibriSpeech, so the data may differ from the distribution of real conversations. RealSSA-Real was recorded with a single mobile device and has no ground-truth speech, so it is evaluated only with reference-free metrics. Within the scope of this text, latency on real edge devices is reported only as GPU latency.

Why it matters

Useful for teams building real-time voice separation on compute-constrained devices such as hearing aids, video conferencing systems and wearables. It also provides an evaluation standard for checking performance in multi-speaker, noisy environments, which fixed two-speaker benchmarks did not reveal.

Key terms

AVTSE (audio-visual target speaker extraction)

A task that uses video of the target speaker's lip movements as a cue to extract only that person's voice from a mixture of several people talking.

Causal model

A model that produces output using only the current and past frames. Because it does not wait for future input, it can be used for streaming.

Near-field and far-field interference

Interfering speakers within about 3 m of the microphone are classed as near-field, and those farther away as far-field. In the far field, reverberant energy exceeds the direct sound.

Contrastive loss

A training objective that pushes the model's output closer to the ground truth and farther from interfering speakers.

Original

• arXiv abstract
• PDF
• HTML full text
• Code
• Project
• Hugging Face

Authors: Wendi Sang, Kai Li, Yifan Li, Jianqiang Huang, Xiaolin Hu · Posted 2026-10-23 · License arXiv 비독점 배포 라이선스 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.