Neuromodulation Gated Transformer

Kobe Knowles, Joshua Bensemann, Diana Benavides-Prado, Vithya Yogarajan, Michael Witbrock, Gillian Dobbie, Yang Chen

Keywords: transformers, neuromodulation, question answering, natural language processing, natural language understanding

TL;DR: This paper introduces a novel approach entwining neuromodulation with transformers. We evaluate performance on the SuperGLUE benchmark and find that it results in the best average performance, although there is variation between datasets.

Abstract: We introduce a novel architecture, the Neuromodulation Gated Transformer (NGT), which is a simple implementation of neuromodulation in transformers via a multiplicative effect. We compare it to baselines and show that it results in the best average performance on the SuperGLUE benchmark validation sets.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/neuromodulation-gated-transformer/code)