LLaSE-G1: Incentivizing Generalization Capability for LLaMA-based Speech Enhancement

ACL ARR 2025 February Submission5225 Authors

Abstract: Recent advancements in language models (LMs) have demonstrated strong capabilities in semantic understanding and contextual modeling, which have flourished in generative speech enhancement (SE). However, many LM-based SE approaches primarily focus on semantic information, often neglecting the critical role of acoustic information, which leads to acoustic inconsistency after enhancement and limited generalization across diverse SE tasks. In this paper, we introduce LLaSE-G1, a LLaMA-based language model that incentivizes generalization capabilities for speech enhancement. LLaSE-G1 offers the following key contributions: First, to mitigate acoustic inconsistency, LLaSE employs continuous representations from WavLM as input and predicts speech tokens from X-Codec2, maximizing acoustic preservation. Second, to promote generalization capability, LLaSE-G1 introduces dual-channel inputs and outputs, unifying multiple SE tasks without requiring task-specific IDs. Third, LLaSE-G1 outperforms prior task-specific discriminative and generative SE models, demonstrating scaling effects at test time and emerging capabilities for unseen SE tasks. Additionally, we release our code and models to support further research in this area.

Paper Type: Long

Research Area: Language Modeling

Research Area Keywords: Language Modeling, Speech Technologies

Contribution Types: Model analysis & interpretability, Reproduction study, Publicly available software and/or pre-trained models

Languages Studied: English, Chinese, Spanish, German, Italian

Submission Number: 5225