BrainAlign: Leveraging EEG Foundation Models for Symmetric, Interpretable Alignment with Visual Representations

Vijay Jayawant Harkare, Lakshmi Subramanian

Keywords: computational neuroscience, machine learning, representation learning, representation alignment, brain-inspired learning, deep learning, foundation models

TL;DR: This study introduces BrainAlign, a contrastive learning framework for interpretable and symmetric alignment of EEG and image representations using pre-trained EEG foundation models.

Abstract: Custom electroencephalography (EEG) encoders trained on limited, task-specific data have restricted ability to learn generalizable, brain-like representations. We propose a representation-first alternative, leveraging a large-scale pretrained EEG foundation model (CBraMod) to learn brain-aligned representations. We introduce BrainAlign, a contrastive learning framework that uses a brain-inspired projection network to align EEG features with those from image encoders. On the challenging 200-way zero-shot visual object classification task, BrainAlign, when paired with a CORNet-S encoder, achieves a top-1 accuracy of 14.2\% and a top-5 accuracy of 37.9\% for EEG-to-image retrieval, performing competitively to prior baselines while reducing training time by 70\%. This computational efficiency is particularly crucial for developing the subject-specific models vital for practical EEG decoding. Additionally, the framework learns a highly symmetric alignment, achieving a 23.2\% top-1 and 54.7\% top-5 accuracy in the reverse image-to-EEG retrieval task. We observe a time-averaged RSA correlation (r = 0.365) with the neuro-inspired CORNet-S model, consistent with a moderately high degree of representational similarity. A post-hoc CCA-INLP analysis isolates a subject-agnostic subspace and, together with a semantic similarity evaluation, shows meaningful category structure yet residual cross-subject variability. Collectively, these results in performance, efficiency, and biological plausibility provide support for our representation-first approach. The resulting robust and symmetric representations can potentially be applicable to demanding downstream applications such as object classification, high-fidelity image decoding directly from brain activity, and real-time object disambiguation.

Primary Area: applications to neuroscience & cognitive science

Submission Number: 23339