Unsupervised Neural Machine Translation

Mikel Artetxe, Gorka Labaka, Eneko Agirre, Kyunghyun Cho

Abstract: In spite of the recent success of neural machine translation (NMT) in standard benchmarks, the lack of large parallel corpora poses a major practical problem for many language pairs. There have been several proposals to alleviate this issue with, for instance, triangulation and semi-supervised learning techniques, but they still require a strong cross-lingual signal. In this work, we completely remove the need of parallel data and propose a novel method to train an NMT system in a completely unsupervised manner, relying on nothing but monolingual corpora. Our model builds upon the recent work on unsupervised embedding mappings, and consists of a slightly modified attentional encoder-decoder model that can be trained on monolingual corpora alone using a combination of denoising and backtranslation. Despite the simplicity of the approach, our system obtains 15.56 and 10.21 BLEU points in WMT 2014 French-to-English and German-to-English translation. The model can also profit from small parallel corpora, and attains 21.81 and 15.24 points when combined with 100,000 parallel sentences, respectively. Our implementation is released as an open source project.

TL;DR: We introduce the first successful method to train neural machine translation in an unsupervised manner, using nothing but monolingual corpora

Keywords: neural machine translation, unsupervised learning

Code: [ rsennrich/subword-nmt](https://github.com/rsennrich/subword-nmt) + [ 1 community implementation](https://paperswithcode.com/paper/?openreview=Sy2ogebAW)

Data: [WMT 2014](https://paperswithcode.com/dataset/wmt-2014), [WMT 2015](https://paperswithcode.com/dataset/wmt-2015)

Community Implementations: [ 4 code implementations](https://www.catalyzex.com/paper/arxiv:1710.11041/code)