The Wisdom of a Crowd of Brains: A Universal Brain Encoder

Roman Beliy, navve wasserman, Amit Zalcher, michal Irani

Keywords: Image-to-fMRI encoding, Explore the brain using ML, Brain-Image cross-attention, Visual Perception, Brain Mapping, Neuroscience, Computer Vision

TL;DR: A Universal Brain-Encoder, which combines data from multiple brains (from many different subjects/datasets/machines, without any shared data).

Abstract:

Image-to-fMRI encoding is important for both neuroscience research and practical applications. However, such “Brain-Encoders” have been typically trained per-subject and per fMRI-dataset, thus restricted to very limited training data. In this paper we propose a Universal Brain-Encoder, which can be trained jointly on data from many different subjects/datasets/machines. What makes this possible is our new voxel-centric Encoder architecture, which learns a unique “voxel-embedding” per brain-voxel. Our Encoder trains to predict the response of each brain-voxel on every image, by directly computing the cross-attention between the brain-voxel embedding and multi-level deep image features. This voxel-centric architecture allows the functional role of each brain-voxel to naturally emerge from
the voxel-image cross-attention. We show the power of this approach to: (i) combine data from multiple different subjects (a “Crowd of Brains”) to improve each individual brain-encoding, (ii) quick & effective Transfer-Learning across sub- jects, datasets, and machines (e.g., 3-Tesla, 7-Tesla), with few training examples, and (iii) we show the potential power of the learned voxel-embeddings to explore brain functionality (e.g., what is encoded where in the brain).

Primary Area: applications to neuroscience & cognitive science

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6247