BrainAlign: Leveraging EEG Foundation Models for Symmetric, Interpretable Alignment with Visual Representations

Vijay Jayawant Harkare, Lakshmi Subramanian

TL;DR: This study introduces BrainAlign, a contrastive learning framework for interpretable and symmetric alignment of EEG and image representations using pre-trained EEG foundation models.

Abstract: Understanding how the human brain represents visual objects is a fundamental challenge that can be addressed by aligning brain activity recordings in the form of electroencephalography (EEG) recordings with features from computer vision models. However, prior work has predominantly relied on custom EEG encoders trained on limited, task-specific data, which restricts their ability to learn generalizable, brain-like representations. In this work, we propose an alternative approach, moving from task-specific encoders to a representation-first approach. We leverage a large-scale pretrained EEG foundation model, CBraMod, to provide a rich and robust foundation for learning brain-aligned representations. We introduce BrainAlign, a contrastive learning framework that uses a brain-inspired projection network to align EEG representations with those from various image encoders (ResNet50, CORNet-S, and CLIP). To evaluate the quality of these aligned representations, we test our framework on the challenging 200-way zero-shot visual object classification task. Using a CORNet-S image encoder, BrainAlign achieves a top-1 accuracy of 14.2\%, exceeding the NICE framework's baseline and performing comparably to state-of-the-art methods that use only vision and EEG modalities. Furthermore, our framework demonstrates significant computational efficiency, reducing the required training epochs by 70\% compared to training from scratch. Moreover, analysis of the learned representational geometry reveals a structure consistent with established phenomena of the human visual system. Collectively, these results in performance, computational efficiency, and biological plausibility validate our representation-first approach, highlighting the potential of foundation models to bridge the gap between neural and artificial representations.

Length: long paper (up to 8 pages)

Domain: methods

Author List Check: The author list is correctly ordered and I understand that additions and removals will not be allowed after the abstract submission deadline.

Anonymization Check: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and URLs that point to identifying information.

Submission Number: 48