Quick Back-Translation for Unsupervised Machine Translation
Submission Type: Regular Long Paper
Submission Track: Machine Translation
Submission Track 2: Natural Language Generation
Keywords: unsupervised machine translation, back-translation, non-autoregressive generation, Transformer
TL;DR: A quick back-translation approach to augment standard back-translation to improve unsupervised machine translation performances efficiently, on top of Transformer models by repurposing the encoder for non-autoregressive generations.
Abstract: The field of unsupervised machine translation has seen significant advancement from the marriage of the Transformer and the back-translation algorithm. The Transformer is a powerful generative model, and back-translation leverages Transformer's high-quality translations for iterative self-improvement. However, the Transformer is encumbered by the run-time of autoregressive inference during back-translation, and back-translation is limited by a lack of synthetic data efficiency. We propose a two-for-one improvement to Transformer back-translation: Quick Back-Translation (QBT). QBT re-purposes the encoder as a generative model, and uses encoder-generated sequences to train the decoder in conjunction with the original autoregressive back-translation step, improving data throughput and utilization. Experiments on various WMT benchmarks demonstrate that a relatively small number of refining steps of QBT improve current unsupervised machine translation models, and that QBT dramatically outperforms standard back-translation only method in terms of training efficiency for comparable translation qualities.
Submission Number: 4744
Loading