EEG2TEXT: Open Vocabulary EEG-to-Text Decoding with EEG Pre-Training and Multi-View Transformer

Hanwen Liu, Daniel Hajialigol, Benny Antony, Aiguo Han, Xuan Wang

Keywords: brain computer interface, eeg, transformer, pretraining

Abstract:

Deciphering the intricacies of the human brain has captivated curiosity for centuries. Recent strides in Brain-Computer Interface (BCI) technology, particularly using motor imagery, have restored motor functions such as reaching, grasping, and walking in paralyzed individuals. However, unraveling natural language from brain signals remains a formidable challenge. Electroencephalography (EEG) is a non-invasive technique used to record electrical activity in the brain by placing electrodes on the scalp. Previous studies of EEG-to-text decoding have achieved high accuracy on small closed vocabularies, but still fall short of high accuracy when dealing with large open vocabularies.
We propose a novel method, EEG2Text, to improve the accuracy of open vocabulary EEG-to-text decoding.
Specifically, EEG2Text leverages EEG pre-training to enhance the learning of semantics from EEG signals and proposes a multi-view transformer to model the EEG signal processing by different t spatial regions of the brain. Experiments show that EEG2Text has superior performance, outperforming the state-of-the-art baseline methods by a large margin of up to 5% in absolute BLEU and ROUGE scores. EEG2Text shows great potential for a high-performance open-vocabulary brain-to-text system to facilitate communication.

Submission Number: 96