NeuroConText: Contrastive Text-to-Brain Mapping for Neuroscientific Literature

Fateme Ghayem, Raphaël Meudec, Jérôme Dockès, Demian Wassermann, Bertrand Thirion

Abstract: Neuroscience faces challenges in reliability due to limited statistical power, reproducibility issues, and inconsistent terminology. To address these challenges, we introduce NeuroConText, the first brain meta-analysis model that uses a contrastive approach to enhance the association between text data and brain activation coordinates reported in 20K neuroscientific articles from PubMed Central. NeuroConText integrates the capabilities of recent large language models (LLMs) rather than traditional bag-of-words methods, to better capture the text semantic, and improve the association with brain activation. It is adapted to processing neuroscientific text regardless of length and generalizes well across various textual content-titles, abstracts, and full-body. Our experiments show NeuroConText significantly outperforms state-of-the-art methods with a threefold increase in linking text to brain activations in terms of recall@10. NeuroConText also allows decoding brain images from latent text representations, successfully maintaining the quality of brain image reconstruction compared to the state-of-the-art.