IBiT: Utilizing Inductive Biases to Create a More Data Efficient Attention Mechanism

ICLR 2026 Conference Submission1773 Authors

Keywords: Computer Vision, Transformers

TL;DR: We used inductive biases that are created through learned masks to create a model that performs significantly better on smaller datasets.

Abstract: In recent years, Transformer-based architectures have become the dominant method for Computer Vision applications. While Transformers are explainable and scale well with dataset size, they lack the inductive biases of Convolutional Neural Networks. While these biases may be learned on large datasets, we show that introducing these inductive biases through learned masks allow Vision Transformers to learn on much smaller datasets without Knowledge Distillation. These Transformers, which we call Inductively Biased Image Transformers (IBiT), are significantly more accurate on small datasets, while retaining the explainability and scalability of Transformers.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 1773