BrainWhisperer: Learning Aligned Semantic Representations from Brain Activity for Language Model-based Decoding

Dongyang Li, Dazhou Liu, Sitong Chen, Jiayu Zuo, Kunpeng Xie, Yiming Liu, Chen Wei, Quanying Liu

Keywords: Brain Decoding, Semantic Decoding, Neurosicence, Foundation Model, LLM, Brain Signals

Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in capturing rich and generalizable semantic representations. In contrast, non-invasive neural signals such as electroencephalography (EEG) lack a well-structured semantic space, making brain-to-text (B2T) decoding especially difficult. This gap motivates us to ask: can neural activity embeddings be aligned with the powerful semantic space of language models, thereby enabling more effective brain decoding? We introduce BrainWhisperer, a novel framework that leverages the rich semantic capabilities of LLMs to address this gap. Our core contribution is an alignment methodology where a Transformer-based encoder, trained on EEG data, is optimized via a contrastive objective to map neural activity into the latent representation space of a powerful, pre-trained and frozen text encoder. This generates unified semantic tokens for language models. We propose and evaluate two decoding pathways: (1) a direct decoding approach where the learned brain embeddings are fed into a lightweight adapter and a frozen text decoder to autoregressively generate text, and (2) an LLM-copilot strategy, where retrieved semantically relevant words from brain embeddings serve as prompts for large language models to generate coherent and context-rich text. Experiments on listening datasets demonstrate that BrainWhisperer produces semantically faithful and fluent text, outperforming baseline approaches. By bridging neural signals with the semantic capacity of LLMs, BrainWhisperer represents a step toward practical and robust brain-to-text communication systems.

Primary Area: applications to neuroscience & cognitive science

Submission Number: 20148