Language decoding from human brain activity via contrastive learning

Matteo Ferrante, Nicola Toschi, Alexander Huth

Track: Extended Abstract Track

Keywords: language, brain decoding, contrastive learning, neural representation of language, sentence identification, fMRI

TL;DR: We present a contrastive learning method to decode language from brain activity using fMRI data. It achieves up to 84% top-10 accuracy in identifying sentences from brain activity by aligning fMRI data with linguistic features in a shared space.

Abstract: We propose a novel contrastive learning approach to decode brain activity into sentences by mapping fMRI recordings and text embeddings into a shared representational space. Using data from three subjects, we trained a cross-subject fMRI encoder and demonstrated effective sentence identification with a retrieval module. Our model shows strong alignment between brain activity and linguistic features, with top-1 accuracy up to 49.2\% and top-10 accuracy up to 84\%, significantly outperforming chance levels. These results highlight the potential of contrastive learning for cross-subject language decoding,

Submission Number: 14