Human-Aligned Image Models Improve Visual Decoding from the Brain

Nona Rajabi, Antonio H. Ribeiro, Miguel Vasco, Farzaneh Taleb, Mårten Björkman, Danica Kragic

TL;DR: This paper proposes leveraging human-aligned image encoders to map brain signals to images using self-supervised learning, achieving up to 21% higher accuracy compared to state of the art.

Abstract: Decoding visual images from brain activity has significant potential for advancing brain-computer interaction and enhancing the understanding of human perception. Recent approaches align the representation spaces of images and brain activity to enable visual decoding. In this paper, we introduce the use of human-aligned image encoders to map brain signals to images. We hypothesize that these models more effectively capture perceptual attributes associated with the rapid visual stimuli presentations commonly used in visual brain data recording experiments. Our empirical results support this hypothesis, demonstrating that this simple modification improves image retrieval accuracy by up to 21\% compared to state-of-the-art methods. Comprehensive experiments confirm consistent performance improvements across diverse EEG architectures, image encoders, alignment methods, participants, and brain imaging modalities.

Lay Summary: Understanding what someone is seeing based on their brain activity has exciting possibilities, such as helping people to communicate without speaking and learning more about how we perceive the world. One way scientists approach this is by linking patterns in brain signals with the visual content of images. In this study, we explore a new method that uses image-processing models trained to perceive images more similarly to humans. These models help translate brain signals into images more accurately, especially during fast-paced visual experiments often used in brain research. Our results show that this approach can improve the accuracy of matching brain signals to images by up to 21% compared to leading methods. We tested our method across different brain recording techniques, participants, and model types, and consistently saw better performance.

Application-Driven Machine Learning: This submission is on Application-Driven Machine Learning.

Link To Code: https://github.com/NonaRjb/AlignVis.git

Primary Area: Applications->Neuroscience, Cognitive Science

Keywords: Visual Decoding, Brain-Computer Interface, EEG, Contrastive Learning, Human-Alignment

Submission Number: 6504