Parrot: Seamless Spoken Dialogue Interaction with Double-Channel Large Language Models

Qichao Wang, Ziqiao Meng, Wenqian Cui, Yifei Zhang, Pengcheng Wu, Bingzhe Wu, Zibin Zheng, Irwin King, Liang Chen, Peilin Zhao

Keywords: Speech Language Models, Generative Spoken Dialogue Language Modeling

TL;DR: We propose a novel spoken dialogue language model that uses an innovative pre-training and SFT pipeline with dual-channel audio data and next-token-pair prediction paradigm, achieving real-time streaming interaction.

Abstract: Recent advancements in large language models (LLMs) have demonstrated significant potential in enhancing real-time spoken interactions. Presently, open-source methodologies predominantly depend on intermediate generative text-based translations to manage real-time spoken dialogues. However, these techniques often struggle with providing seamless interactions that involve real-time streaming audio inputs. In this research, we unveil an innovative spoken dialogue language model, Parrot, distinguished by its unique pre-training and supervised fine-tuning (SFT) pipeline. This pipeline deviates from conventional methodologies by utilizing both single-channel audio data and double-channel spoken dialogue data to train the textless speech language model. During pre-training, we transmute single-channel audio input into a sequence of discrete tokens, thereby instructing the LLM to identify audio tokens via next-token predictions. In the SFT phase, we pioneer a novel approach to double-channel generative spoken dialogue language modeling with a unique ``next-token-pair prediction" objective, facilitating the LLM's comprehension of natural human conversations. Our inventive pipeline equips the LLM to produce spoken interactions that are more natural and fluid than those generated by previous text-based approaches, as substantiated by thorough evaluations.

Primary Area: foundation or frontier models, including LLMs

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6581