SSR: Alignment-Aware Modality Connector for Speech Language Models

Weiting Tan, Hirofumi Inaguma, Ning Dong, Paden Tomasello, Xutai Ma

Keywords: modality fusion, speech language model, spoken language understanding

Abstract: Fusing speech into pre-trained language model (SpeechLM) usually suffers from inefficient encoding of long-form speech and catastrophic forgetting of pre-trained text modality. We propose SSR-connector (Segmented Speech Representation Connector) for better modality fusion. Leveraging speech-text alignments, our approach segments and compresses speech features to match the granularity of text embeddings. Additionally, we introduce a two-stage training pipeline that includes the distillation and fine-tuning phases to mitigate catastrophic forgetting. SSR-connector outperforms existing mechanism for speech-text modality fusion, consistently achieving better speech understanding (e.g., $+10$ accuracy on StoryCloze and $+20$ on Speech-MMLU) while preserving pre-trained text ability.

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7695