Attention Approximates Sparse Distributed MemoryDownload PDF

Published: 09 Nov 2021, Last Modified: 22 Oct 2023NeurIPS 2021 PosterReaders: Everyone
Keywords: Transformer, Attention, Sparse Distributed Memory, Associative Memory, Cerebellum
Abstract: While Attention has come to be an important mechanism in deep learning, there remains limited intuition for why it works so well. Here, we show that Transformer Attention can be closely related under certain data conditions to Kanerva's Sparse Distributed Memory (SDM), a biologically plausible associative memory model. We confirm that these conditions are satisfied in pre-trained GPT2 Transformer models. We discuss the implications of the Attention-SDM map and provide new computational and biological interpretations of Attention.
Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.
Supplementary Material: pdf
TL;DR: We show the heuristic Transformer Attention operation can be implemented with simple properties of high dimensional vectors, in a biologically plausible fashion using SDM.
Community Implementations: [![CatalyzeX](/images/catalyzex_icon.svg) 3 code implementations](
13 Replies