Gradient-based Gradual Pruning for Language-Specific Multilingual Neural Machine Translation
Submission Type: Regular Long Paper
Submission Track: Machine Translation
Submission Track 2: Multilinguality and Linguistic Diversity
Keywords: Multilingual neural machine translation, language-specific sub-network extraction, model pruning, gradual pruning schedule, gradient-based pruning criterion
TL;DR: Applying gradient-based gradual pruning for language-specific multilingual neural machine translation.
Abstract: Multilingual neural machine translation (MNMT) offers the convenience of translating between multiple languages with a single model. However, MNMT often suffers from performance degradation in high-resource languages compared to bilingual counterparts. This degradation is commonly attributed to parameter interference, which occurs when parameters are fully shared across all language pairs. In this work, to tackle this issue we propose a gradient-based gradual pruning technique for MNMT. Our approach aims to identify an optimal sub-network for each language pair within the multilingual model by leveraging gradient-based information as pruning criterion and gradually increasing the pruning ratio as schedule. Our approach allows for partial parameter sharing across language pairs to alleviate interference, and each pair preserves its unique parameters to capture language-specific information. Comprehensive experiments on IWSLT and WMT datasets show that our approach yields a notable performance gain on both datasets.
Submission Number: 3445
Loading