Posterior Attention Models for Sequence to Sequence Learning

Shiv Shankar, Sunita Sarawagi

Abstract:

Modern neural architectures critically rely on attention for mapping structured inputs to sequences. In this paper we show that prevalent attention architectures do not adequately model the dependence among the attention and output tokens across a predicted sequence.
We present an alternative architecture called Posterior Attention Models that after a principled factorization of the full joint distribution of the attention and output variables, proposes two major changes. First, the position where attention is marginalized is changed from the input to the output. Second, the attention propagated to the next decoding stage is a posterior attention distribution conditioned on the output. Empirically on five translation and two morphological inflection tasks the proposed posterior attention models yield better BLEU score and alignment accuracy than existing attention models.

Keywords: posterior inference, attention, seq2seq learning, translation

TL;DR: Computing attention based on posterior distribution leads to more meaningful attention and better performance