Monotonic Multihead Attention

Xutai Ma, Juan Miguel Pino, James Cross, Liezl Puzon, Jiatao Gu

TL;DR: Make the transformer streamable with monotonic attention.

Abstract: Simultaneous machine translation models start generating a target sequence before they have encoded or read the source sequence. Recent approach for this task either apply a fixed policy on transformer, or a learnable monotonic attention on a weaker recurrent neural network based structure. In this paper, we propose a new attention mechanism, Monotonic Multihead Attention (MMA), which introduced the monotonic attention mechanism to multihead attention. We also introduced two novel interpretable approaches for latency control that are specifically designed for multiple attentions. We apply MMA to the simultaneous machine translation task and demonstrate better latency-quality tradeoffs compared to MILk, the previous state-of-the-art approach.

Keywords: Simultaneous Translation, Transformer, Monotonic Attention

Code: [ pytorch/fairseq](https://github.com/pytorch/fairseq) + [ 2 community implementations](https://paperswithcode.com/paper/?openreview=Hyg96gBKPS)

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:1909.12406/code)