On the Off-Target Problem of Zero-Shot Multilingual Neural Machine Translation

Anonymous

Abstract: While multilingual neural machine translation has achieved great success, it suffers from the off-target issue, where the translation is in the wrong language. This problem is more pronounced on zero-shot translation tasks. In this work, we explore the major cause of the off-target problem and find that a closer lexical distance (i.e., KL-divergence) between two languages' vocabularies leads to a higher off-target rate. Motivated by the finding, we propose LAVS, a simple and effective algorithm to construct the multilingual vocabulary, that greatly alleviates the off-target problem of the translation model by increasing the KL-divergence between languages. We conduct experiments on a multilingual machine translation benchmark in 11 languages. Experiments show that the off-target rate for 81 translation tasks is reduced from 29\% to 8\%, while the overall BLEU score is improved by an average of 1.9 points. We will release the code for reproducibility.

Paper Type: long

Research Area: Machine Translation