Enhancing the Transformer with explicit relational encoding for math problem solving

Imanol Schlag, Paul Smolensky, Roland Fernandez, Nebojsa Jojic, Jürgen Schmidhuber, Jianfeng Gao

TL;DR: Our Tensor-Product Transformer sets a new state of the art on the recently-introduced Mathematics Dataset containing 56 categories of free-form math word-problems.

Abstract:

We incorporate Tensor-Product Representations within the Transformer in order to better support the explicit representation of relation structure.
Our Tensor-Product Transformer (TP-Transformer) sets a new state of the art on the recently-introduced Mathematics Dataset containing 56 categories of free-form math word-problems.
The essential component of the model is a novel attention mechanism, called TP-Attention, which explicitly encodes the relations between each Transformer cell and the other cells from which values have been retrieved by attention. TP-Attention goes beyond linear combination of retrieved values, strengthening representation-building and resolving ambiguities introduced by multiple layers of regular attention.
The TP-Transformer's attention maps give better insights into how it is capable of solving the Mathematics Dataset's challenging problems.
Pretrained models and code will be made available after publication.

Keywords: Tensor Product Representation, Transformer, Mathematics Dataset, Attention

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:1910.06611/code)