EEG-Language Pretraining for Highly Label-Efficient Pathology Detection

Sam Gijsen, Kerstin Ritter

Keywords: deep learning, multimodal, neuroscience, eeg, medical

TL;DR: A first application of multimodal language pretraining to the domain of medical EEG to improve pathology detection.

Abstract: Multimodal language modeling constitutes a recent breakthrough which leverages advances in large language models to pretrain capable multimodal models. The integration of natural language during pretraining has been shown to significantly improve learned representations, particularly in computer vision. However, the efficacy of multimodal language modeling in the realm of functional brain data, specifically for advancing pathology detection, remains unexplored. This study pioneers EEG-language models (ELMs) trained on clinical reports and 15000 EEGs. We propose to combine multimodal alignment in this novel domain with timeseries cropping and text segmentation. This also enables an extension based on multiple instance learning to alleviate misalignment between irrelevant EEG or text segments. Our results indicate that models learn richer representations from being exposed to a variety of report segments, including the patient's clinical history, description of the EEG, and the physician's interpretation. Compared to models exposed to narrower clinical text information, we find such models to retrieve EEGs based on clinical reports (and vice versa) with substantially higher accuracy. Particularly in regimes with few annotations, we observe that ELMs can significantly improve pathology detection compared to EEG-only models, as demonstrated by both zero-shot classification and linear probes. The integration of multiple instance learning further improves performance across tasks. In sum, these results highlight the potential of integrating brain activity data with clinical text, suggesting that ELMs represent significant progress for clinical applications.

Primary Area: applications to neuroscience & cognitive science

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7220