Language-Family Adapters for Multilingual Neural Machine Translation

Anonymous

Abstract: Massively multilingual pretrained models yield state-of-the-art results in a wide range of cross-lingual natural language processing tasks. For machine translation, the de facto way to leverage knowledge of pretrained models is fine-tuning on parallel data from one or multiple language pairs. Multilingual fine-tuning improves performance on medium- and low-resource languages but requires modifying the entire model and can be prohibitively expensive. Training either language-pair specific or language-agnostic adapters while keeping most of the pretrained model's parameters frozen has been proposed as a lightweight alternative. However, the former do not learn useful cross-lingual representations for multiple language pairs, while the latter share parameters for all languages and potentially have to deal with negative interference. In this paper, we propose training language-family adapters on top of a pretrained multilingual model to facilitate cross-lingual transfer. Using language families, our model consistently outperforms other adapter-based approaches and is on par with multilingual fine-tuning, while being more efficient. We also demonstrate that language-family adapters provide an effective method to translate to languages unseen during pretraining and substantially outperform the baselines.