Towards human-like spoken dialogue generation between AI agents from written dialogue

Kentaro Mitsui, Yukiya Hono, Kei Sawada

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: spoken dialogue modeling, text-to-speech synthesis, backchannel generation, turn-taking

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: Building a system that generates spoken dialogues with natural backchannels and enables smooth turn-taking from written dialogues.

Abstract:

The advent of large language models (LLMs) has made it possible to generate natural written dialogues between two agents.
However, generating human-like spoken dialogues from these written dialogues remains challenging.
Spoken dialogues have several unique characteristics: they frequently include backchannels and laughter, and the smoothness of turn-taking significantly influences the fluidity of conversation.
This study proposes CHATS ― CHatty Agents Text-to-Speech ― a discrete token-based system designed to generate spoken dialogues based on written dialogues.
Our system can generate speech for both the speaker side and the listener side simultaneously, using only the transcription from the speaker side, which eliminates the need for transcriptions of backchannels or laughter.
Moreover, CHATS facilitates natural turn-taking; it determines the appropriate duration of silence after each utterance in the absence of overlap, and it initiates the generation of overlapping speech based on the phoneme sequence of the next utterance in case of overlap.
Experimental evaluations indicate that CHATS outperforms the text-to-speech baseline, producing spoken dialogues that are more interactive and fluid while retaining clarity and intelligibility.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3338