Bridging Vision, Language, and Brain: Whole-Brain Interpretation of Visual Representations via Information Bottleneck Attribution

Haoyu Li, Hao Wu, Liangjun Chen, Badong Chen

Keywords: Brain Decoding, Whole Brain Model, Brain Attribution, Information Bottleneck, Contrastive Learning

TL;DR: We bridge brain activity with visual and linguistic by proposing a whole-brain representation model and, building on this alignment, develop a tri-modal information bottleneck brain attribution to study cortical visual representations.

Abstract: Understanding how the human brain processes and integrates visual and linguistic information is a long-standing challenge in both cognitive neuroscience and artificial intelligence. In this work, we present two contributions toward attributing visual representations in the cortex by bridging brain activity with natural modalities. We first align fMRI signals with image and text embeddings from a pre-trained CLIP model by proposing a whole-brain representation module that follows anatomical alignment, preserves voxel spatial topology, and captures distributed brain dynamics. Building on this foundation, we further develop an Information Bottleneck-based Brain Attribution (IB-BA) method, which extends information-theoretic attribution to a tri-modal setting. IB-BA identifies the most informative subset of voxels for visual tasks by maximizing mutual information with image and text embeddings while enforcing compression relative to perturbed brain features. Experiments demonstrate superior cross-modal retrieval performance and yield more interpretable cortical attribution maps compared to existing approaches. Collectively, our findings point to new directions for linking neural activity with multimodal representations.

Primary Area: applications to neuroscience & cognitive science

Submission Number: 23013