Condensing Multilingual Knowledge with Lightweight Language-Specific Modules

Haoran Xu, Weiting Tan, Shuyue Stella Li, Yunmo Chen, Benjamin Van Durme, Philipp Koehn, Kenton Murray

Submission Type: Regular Long Paper

Submission Track: Machine Translation

Submission Track 2: Multilinguality and Linguistic Diversity

Keywords: Multilingual Machine Translation, Lightweight, Language interference, Distillation

TL;DR: We propose lightweight language-specific parameters to boost multilingual performance and parameter efficiency, and propose fuse distillation to further improve the efficiency of inference and model serialization.

Abstract: Incorporating language-specific (LS) modules or Mixture-of-Experts (MoE) are proven methods to boost performance in multilingual model performance, but the scalability of these approaches to hundreds of languages or experts tends to be hard to manage. We present Language-specific Matrix Synthesis (LMS), a novel method that addresses the issue. LMS utilizes parameter-efficient and lightweight modules, reducing the number of parameters while outperforming existing methods, e.g., +1.73 BLEU over Switch Transformer on OPUS-100 multilingual translation. Additionally, we introduce Fuse Distillation (FD) to condense multilingual knowledge from multiple LS modules into a single shared module, improving model inference and storage efficiency. Our approach demonstrates superior scalability and performance compared to state-of-the-art methods.

Submission Number: 2444