Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM

We present a novel approach to adapting pre-trained large language models (LLMs) to perform question answering (QA) and speech continuation. By endowing the LLM with a pre-trained speech encoder, our model becomes able to take speech inputs and generate speech outputs. The entire system is trained end-to-end and operates directly on spectrograms, simplifying our architecture. Key to our approach is a training objective that jointly supervises speech recognition, text continuation, and speech synthesis using only paired speech-text pairs, enabling a `cross-modal' chain-of-thought within a single decoding pass. Our method surpasses existing spoken language models in speaker preservation and semantic coherence. Furthermore, the proposed model improves upon direct initialization in retaining the knowledge of the original LLM as demonstrated through spoken QA datasets.

Model

The first audio column labeled Prompt plays the original audio prompt, which is the first 3 seconds of an example from the LibriSpeech test-clean test set. The following column, Spectron, plays the continuation generated by our model for the prompt appearing in the first column.The next columns GSLM, AudioLm, TWIST and SpeechGPT contain the GSLM, AudioLM, TWIST, SpeechGPT continuation for the same prompt repsectivly. For AudioLM we present the 12RVQ model and For TWIST the 7B model.

Librispeech continuation

PromptSpectronGSLMAudioLmTWISTSpeechGPT

The first audio column labeled Question plays a question from the LLama QA dataset we had created. The following column, Spectron, plays the answer generated by our model for the question appearing in the first column. The next columns GSLM, AudioLm, TWIST and SpeechGPT contain the GSLM, AudioLM, TWIST, SpeechGPT answeres for the same question repsectivly. For AudioLM we present the 12RVQ model and For TWIST the 7B model.

LLama questions

QuestionSpectronGSLMAudioLMTWISTSpeechGPT

The first audio column labeled Question plays a question from the Spoken WebQuestions dataset. The following column, Spectron, plays the answer generated by our model for the question appearing in the first column. The next columns GSLM, AudioLm, TWIST and SpeechGPT contain the GSLM, AudioLM, TWIST, SpeechGPT answeres for the same question repsectivly. For AudioLM we present the 12RVQ model and For TWIST the 7B model.

Spoken WebQuestions

QuestionSpectronGSLMAudioLMTWISTSpeechGPT