Large EEG-U-Transformer for Time-Step-Level Detection Without Pre-Training

Kerui Wu, Ziyue Zhao, Bulent Yener

Keywords: EEG, AI for Science, Deep Learning, Seizure Detection

Abstract: Electroencephalography (EEG) reflects the brain's functional state, making it a crucial tool for diverse detection applications, including event-centric analysis like seizure detection and status-centric analysis like pathological detection. While deep learning-based approaches have recently shown promise for automated detection, traditional models are often constrained by limited learnable parameters and only achieve modest performance. In contrast, large foundation models showed improved capabilities by scaling up the model size, but required extensive time-consuming pre-training. Moreover, both types of existing methods focus on window-level classification, which requires redundant post-processing pipelines for event-centric tasks. In this work, based on the multi-scale nature of EEG events, we propose a simple U-shaped model to efficiently learn representations by capturing both local and global features using convolution and self-attentive modules for sequence-to-sequence modeling. Compared to other window-level classification models, our method directly outputs predictions at the time-step level, eliminating redundant overlapping inferences. Beyond sequence-to-sequence modeling, the architecture naturally extends to window-level classification by incorporating an attention-pooling layer. Such a paradigm shift and model design demonstrated promising efficiency improvement, cross-subject generalization, and state-of-the-art performance in various time-step and window-level classification tasks in the experiment. More impressively, our model showed the capability to be scaled up to the same level as existing large foundation models that have been extensively pre-trained over diverse datasets and outperforms them by solely using the downstream fine-tuning dataset.

Primary Area: applications to neuroscience & cognitive science

Submission Number: 15972