Thinking of LLM as a Brain: Decoding Text by Aligning Semantic in EEG Signals and Language

ACL ARR 2025 February Submission609 Authors

Abstract:

With the rapid development of brain-computer interfaces (BCI) in recent years, the electroencephalography (EEG) to text task has drawn increasing attention.
Previous methods have defined it as a sequence-to-sequence translation task. However, their models, trained using teacher-forcing strategies, fail to extract and utilize EEG information.
To address this issue, we propose a novel framework in this paper, which innovatively treats the EEG-to-text task as a fine-grained controllable text generation task.
Specifically, since large language models (LLMs) have strong text generation capabilities, we treat the LLM as a "brain" and guide it to generate desired sentences by leveraging EEG representations that are aligned with the semantic space of text.
Therefore, our approach focuses on training an EEG representation model that can effectively align EEG representation with text semantics, avoiding the limitations introduced by teacher-forcing strategies.
Extensive experiments on the ZuCo benchmark demonstrate the effectiveness of our approach, which achieves state-of-the-art performance in multi-subject and single-subject settings. Furthermore, experimental results in cross-subject scenarios further verify that our method has a strong generalization ability and can be applied to unseen subjects.

Paper Type: Long

Research Area: Generation

Research Area Keywords: data-to-text generation, interactive and collaborative generation

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Publicly available software and/or pre-trained models

Languages Studied: English

Submission Number: 609