A Novel Two-step Fine-tuning Framework for Transfer Learning in Low-Resource Neural Machine Translation

Anonymous

TL;DR: Our paper introduces a novel transfer learning framework that first adjusts the parent model parameters to fit the child language and then transferring the adjusted parameters to the child model.

Abstract: Existing transfer learning methods for neural machine translation typically use a well-trained translation model (i.e., a parent model) of a high-resource language pair to directly initialize a translation model (i.e., a child model) of a low-resource language pair, and the child model is then fine-tuned with corresponding datasets. In this paper, we propose a novel two-step fine-tuning (TSFT) framework for transfer learning in low-resource neural machine translation. In the first step, we adjust the parameters of the parent model to fit the child language by using the child monolingual data. In the second step, we transfer the adjusted parameters to the child model and fine-tune it with a proposed distillation loss for efficient optimization. Our experimental results on five low-resource translations demonstrate that our framework yields significant improvements over various strong transfer learning baselines.

Paper Type: long

Research Area: Machine Translation

Contribution Types: Approaches to low-resource settings

Languages Studied: En, De, Tr, Hu, Id, Ca, Pl