BELT-2: Bootstrapping EEG-to-Language representation alignment for multi-task brain decoding

Charles Chau, Yiqun Duan, Yu-Cheng Chang, Thomas Do, Yu-Kai Wang, Chin-teng Lin

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: EEG, Brain, Decoding

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: The first state-of-the-art multi-task brain decoding model

Abstract: The remarkable success of large language models (LLMs) across various multi-modality applications is evident. However, integrating large language models with humans, or brain dynamics, remains relatively unexplored. In this paper, we introduce BELT-2, a pioneering multi-task model designed to enhance both encoding and decoding performance from EEG signals. To bolster the quality of the EEG encoder, BELT-2 is the first work to innovatively 1) adopt byte pair encoding (BPE)-level EEG-language alignment and 2) integrate multi-task training and decoding in the EEG domain. Inspired by the idea of Bridging the Brain with GPT, we connect the multi-task EEG encoder with LLMs by utilizing prefix-tuning on intermediary output from the EEG encoder. These remarkable advancements firmly establish BELT-2 as a pioneering breakthrough, making it the first work in the field capable of decoding coherent and readable sentences from non-invasive brain signals. Our experiments highlight significant advancements over prior techniques in both quantitative and qualitative measures, achieving a remarkable decoding performance with a BLEU-1 score of 52.2% on the ZuCo dataset. Furthermore, BELT-2 shows an improvement ranging from 31% to 162% on other translation benchmarks. Codes can be accessed via the provided anonymous link.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2330