Systematic Generalization with Edge Transformers

Leon Bergen, Timothy J. O'Donnell, Dzmitry Bahdanau

Keywords: systematic generalization, transformer, semantic parsing, language understanding

TL;DR: We propose a new model that is inspired by transformers and logical reasoning. The proposed model generalizes more systematically.

Abstract: Recent research suggests that systematic generalization in natural language understanding remains a challenge for state-of-the-art neural models such as Transformers and Graph Neural Networks. To tackle this challenge, we propose Edge Transformer, a new model that combines inspiration from Transformers and rule-based symbolic AI. The first key idea in Edge Transformers is to associate vector states with every edge, that is, with every pair of input nodes---as opposed to just every node, as it is done in the Transformer model. The second major innovation is a triangular attention mechanism that updates edge representations in a way that is inspired by unification from logic programming. We evaluate Edge Transformer on compositional generalization benchmarks in relational reasoning, semantic parsing, and dependency parsing. In all three settings, the Edge Transformer outperforms Relation-aware, Universal and classical Transformer baselines.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

Code: https://github.com/bergen/EdgeTransformer

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2112.00578/code)