A Neuroscience-inspired Framework for Tri-modality Alignment of Brain Signals, Vision, and Language

Yixing Ke, Binghao Ye, Dong Liang, Kun Shang

Keywords: Brain-Computer Interface, Neuroscience-Inspired Alignment, Dynamic Semantic Guidance, Visual Retrieval

Abstract: Visual retrieval from brain signals is a key challenge in Brain-Computer Interfaces (BCIs). Existing methods mainly rely on direct cross-modality mapping, yet they often overlook the neural mechanisms of visual processing, which leads to three major limitations. First, a feature physiology mismatch arises because high-level semantic features extracted by image encoders do not align with the low-level neural responses evoked by rapid visual stimulation. Second, most approaches emphasize cross-modality alignment while neglecting the similarity of neural representations within the same category, which results in poor intra-modality semantic consistency. Third, brain-image alignment typically depends on static image-text semantic spaces and therefore lacks dynamic semantic priors that interact with brain activity. We introduce NeuroAlign, the first neuroscience-inspired framework for brain-vision alignment. NeuroAlign mitigates the feature physiology mismatch by integrating bottom-up structural perception with top-down semantic modulation, enhances semantic consistency through intra-modality self-supervision and cross-modality intra-class constraints, and leverages large language models (LLMs) to provide dynamic semantic signals that interact dynamically with brain responses. Extensive experiments demonstrate that NeuroAlign achieves state-of-the-art performance on both intra-subject and inter-subject retrieval tasks, which validates the effectiveness of this neuroscience-guided alignment strategy.

Primary Area: applications to neuroscience & cognitive science

Submission Number: 3314