Learn and Consolidate: Continual Adaptation for Zero-Shot and Multilingual Neural Machine Translation

Kaiyu Huang, Peng Li, Junpeng Liu, Maosong Sun, Yang Liu

Submission Type: Regular Long Paper

Submission Track: Machine Translation

Keywords: Multilingual Neural Machine Translation, Zero-shot Machine Translation, Continual Learning

TL;DR: We effectively promote both supervised and zero-shot performance for multilingual neural machine translation model using newly available data while maintaining original stronger performance on English-Centric directions for continual adaptation.

Abstract: Although existing multilingual neural machine translation (MNMT) models have demonstrated remarkable performance to handle multiple translation directions in a single model and achieved zero-shot translation between language pairs unseen in training, they still suffer from relatively poor translation qualities for some language pairs. A practical scenario is that how to continually update MNMT models for both supervised and zero-shot translations when limited new data arrives. To this end, we propose a two-stage approach that encourages original models to acquire language-agnostic multilingual representations from new data, and preserves the model architecture without introducing parameters. Experimental results and further analysis demonstrate that our method can efficiently improve performance of existing MNMT models in translation directions where they are initially weak, and mitigates the degeneration in the original well-performing translation directions, offering flexibility in the real-world scenario.

Submission Number: 5048