FastMSS: A Scalable Synthetic Conversations Generator

Alexander Polok, Ivan Medennikov, Shinji Watanabe, Jan Cernocký, Lukáš Burget, Samuele Cornell

Keywords: multi-talker speech processing, speaker diarization, target-speaker automatic speech recognition, synthetic data

TL;DR: Synthetic conversational data can nearly match real training data for multi-talker ASR and speaker diarization, but the optimal simulation recipe differs by task, and combining both is best.

Abstract: Recent breakthroughs in multi-talker ASR (MT-ASR) and speaker diarization (SD) rely on synthetic data to mitigate the scarcity of large-scale conversational recordings, yet the impact of specific simulation choices remains poorly understood. To mind the gap between simulated mixtures and real-world interactions, we present a study of synthetic data generation for leading MT-ASR (DiCoW) and SD (Sortformer) systems. By introducing FastMSS, a highly efficient open-source simulator, we analyze turn-taking dynamics, source domain, acoustic augmentation, and data mixing strategies. Our findings reveal that optimal simulation recipes are highly task-dependent: increasing speech overlap benefits ASR but degrades diarization. Furthermore, broad source diversity consistently outperforms exact domain matching. Ultimately, synthetic-only training approaches real-data baselines, and combining simulated data with real recordings yields substantial gains over real-only training across both tasks.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 157