AV-Dialog: Spoken Dialogue Models with Audio-Visual Input

ACL ARR 2026 January Submission10597 Authors

Keywords: spoken dialogue systems, multi-modal dialogue systems

Abstract: Dialogue models falter in noisy, multi-speaker environments, often producing irrelevant responses and awkward turn-taking. We present AV-Dialog, the first multimodal dialog framework that uses both audio and visual cues to track the target speaker, predict turn-taking, and generate coherent responses. By combining acoustic tokenization with multi-task, multi-stage training on monadic, synthetic, and real audio-visual dialogue datasets, AV-Dialog achieves robust streaming transcription, semantically grounded turn-boundary detection and accurate responses, resulting in a natural conversational flow. Experiments show that AV-Dialog outperforms audio-only models under interference, reducing transcription errors, improving turn-taking prediction, and enhancing human-rated dialogue quality. These results highlight the power of seeing as well as hearing for speaker-aware interaction, paving the way for spoken dialogue agents that perform robustly in real-world, noisy environments.

Paper Type: Long

Research Area: Dialogue and Interactive Systems

Research Area Keywords: spoken dialogue systems, multi-modal dialogue systems

Contribution Types: NLP engineering experiment

Languages Studied: English

Submission Number: 10597