NeuroFusion: A Unified Framework for Generalized Visual Stimulus Decoding from fMRI Across Datasets and Subjects

Muhammad Kashif, Matteo Ferrante, Nicola Toschi

Track: Extended Abstract Track

Keywords: Brain decoding, Computational neuroscience, Neuroimaging, Neuroscience

TL;DR: We present a unified brain decoding framework that generalizes across subjects and datasets by aligning fMRI signals with image embeddings via contrastive learning and enhancing robustness through biologically plausible data augmentation.

Abstract: Recent advancements in neural decoding have shown promising results in reconstructing visual experiences from brain activity. However, existing approaches focus primarily on decoding within a single dataset or subject, which limits generalization across various sources of neuroimaging. In this work, we propose a novel framework for the decoding of visual stimuli \textbf{ between subjects and between data sets}, integrating neural recordings from multiple publicly available fMRI datasets. To address inherent intersubject and interdataset variability, we introduce a contrastive learning-based alignment strategy using image embeddings from a pre-trained IP-Adapter model. Our approach learns a shared latent space by aligning subject-specific neural representations with image features, enabling generalized decoding across both subjects and datasets. In addition, we propose a simple yet effective data augmentation method using ridge regression. This method synthesizes realistic fMRI-like signals from novel images by predicting voxel activity and injecting learned noise distributions, thus enhancing training diversity and model robustness. To the best of our knowledge, while several recent studies have explored cross-subject decoding, our work is the first to extend this direction to joint decoding across multiple datasets and subjects using a unified training framework. Empirical results show that our method achieves competitive and, in some metrics, state-of-the-art decoding performance in this more challenging and realistic setting.

Submission Number: 33