Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep LearningDownload PDF

21 May 2021, 20:46 (edited 15 Jan 2022)NeurIPS 2021 PosterReaders: Everyone
  • Keywords: attention, self-attention, transformers, multi-head self-attention, dot-product attention, equivariant, equivariance, invariant, invariance, interactions, tabular, supervised learning, masking
  • TL;DR: We introduce a novel deep learning architecture that takes the entire dataset as input and learns to reason about relationships between datapoints using self-attention.
  • Abstract: We challenge a common assumption underlying most supervised deep learning: that a model makes a prediction depending only on its parameters and the features of a single input. To this end, we introduce a general-purpose deep learning architecture that takes as input the entire dataset instead of processing one datapoint at a time. Our approach uses self-attention to reason about relationships between datapoints explicitly, which can be seen as realizing non-parametric models using parametric attention mechanisms. However, unlike conventional non-parametric models, we let the model learn end-to-end from the data how to make use of other datapoints for prediction. Empirically, our models solve cross-datapoint lookup and complex reasoning tasks unsolvable by traditional deep learning models. We show highly competitive results on tabular data, early results on CIFAR-10, and give insight into how the model makes use of the interactions between points.
  • Supplementary Material: pdf
  • Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.
  • Code:
20 Replies