Temporal Attention Modules for Memory-Augmented Neural Networks

Rodolfo Palma, Alvaro Soto, Luis Martí, Nayat Sanchez-pi

Keywords: multitasking, attention, deep learning, natural language processing

Abstract:

We introduce two temporal attention modules which can be plugged into traditional memory augmented recurrent neural networks to improve their performance in natural language processing tasks.
The temporal attention modules provide new inductive biases allowing the models to compute attention distributions over the different time steps of input sequences.
The values of these attention distributions can be inspected to identify the sequence's elements that the model considered relevant during the inference.
Using the Entity Network (Henaff et al., 2016) as the model backbone, experiments were made on the dataset bAbI tasks, a set of QA tasks.
Due to the addition of the temporal attention modules, the performance metric increased 26% when the temporal attention was supervised, and 13,5% when it wasn't.
Moreover, the usage of temporal attention modules proved useful at resolving reasoning tasks that the original model was unable to solve.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: Temporal attention modules are useful at enriching memory--augmented neural networks and improve their performance

Reviewed Version (pdf): https://openreview.net/references/pdf?id=Xog3mH1IB6a