Linear Maps, Contrastive Objectives: A Principled Strategy for fMRI Decoding Consistent Across Modalities

ICLR 2026 Conference Submission17992 Authors

Keywords: Computational Neuroscience, Cognitive Science, fMRI Decoding, Contrastive Learning

TL;DR: Linear contrastive alignment is the most effective way to decode fMRI activity, by leveraging vectorial representations of concepts and the linearization inherent to fMRI measurements.

Abstract: A prominent theory in cognitive science suggests that concepts in the brain are organized as high-dimensional vectors, with semantic meaning captured by directions and relative angles in this space. Brain decoding is the effort of reconstructing or retrieving stimuli (or their representations) from neural activity and involves finding a function that approximates how the brain represents concepts. This motivates the investigation of contrastive objectives as biologically plausible candidates to reverse the brain loss function. In this work, we study how functional MRI (fMRI) activity can generally be aligned with the embedding spaces of foundation models in vision, language, and audio. Although neural computations are highly non-linear at the microscale, fMRI measurements average signals across space and time, further smoothed by noise, effectively linearizing the observable representation. Consistent with these views, our experiments across multiple datasets demonstrate that linear contrastive decoders consistently outperform ridge regression and non-linear alternatives, and that these results generalize across images, text, and sound. These findings indicate that decoding gains arise more from the choice of training objective than from architectural complexity, pointing to contrastive-linear models as a principled strategy for brain decoding.

Primary Area: applications to neuroscience & cognitive science

Submission Number: 17992