BrainMIND: Interpret Fine-grained Spatial Mapping of Brain Activity to Multi-semantic Concepts

Zicong He, ShiRunze, Tianxing He, Lu Mi

Keywords: computational neuroscience, visual cortex, fmri, mixselectivity, brain map, brain decoding

TL;DR: We provide an interpret fine-grained spatial mapping of brain activity to multi-semantic concepts, and explore semantic organization in the higher visual cortex.

Abstract: Understanding how population-coding in the human visual cortex shape high-level semantic representations remains a significant challenge. Prior work has either focused on region-level text decoding or relied on simple linear models to probe single-semantic decoding at the voxel level. Consequently, systematic exploration of semantic diversity remains limited at both the region level and the fine-grained voxel level. To address this gap, we introduce BrainMIND, a data-driven framework for analyzing multi-concept semantic selectivity in the visual cortex. We use a conditional variational autoencoder (CVAE) whose latent space is constrained by brain data and spatial locations of voxels. The CVAE decodes the structured latent space into CLIP-aligned semantic embeddings, which then condition a fine-tuned large language model to generate interpretable captions. We validate BrainMIND on widely recognized cortical regions, demonstrating interpretable region-level and voxel-level semantic selectivity. We reveal that individual voxels exhibit mixed selectivity across multiple semantic dimensions, and filling a key gap in voxel-wise neural decoding. Our results demonstrate that BrainMIND provides an interpretable bridge from brain regions to their constituent voxels, enabling controlled, fine-grained exploration of semantic organization in the higher visual cortex.

Primary Area: applications to neuroscience & cognitive science

Submission Number: 24263