Learning Space-Time-Frequency Representation with Two-Stream Attention Based 3D Network for Motor Imagery Classification

Zhenqi Li, Jing Wang, Ziyu Jia, Youfang Lin

Abstract: Motor imagery (MI), as one of the important applications of brain-computer interface (BCI), has lately received great attention. However, current MI researches have not provided satisfactory representations of electroencephalogram (EEG), taking account of the space-time-frequency features for MI classification. Moreover, those models also lack the exploration of attentive spatial, temporal, and spectral dynamics. In this study, we propose TA3D (Two-stream Attention based 3D network), a novel model for MI classification. It mainly consists of two streams: the space-time stream and the space-frequency stream, representing and learning discriminative features in the space-time-frequency dimension. Specifically, each stream contains three key parts: 1) 3D representations of EEG signals depict the spatial information over temporal/spectral distributions; 2) Attention mechanisms adaptively explore attentive dynamics of EEG signals and focus on the most valuable information in separate dimensions; 3) 3D convolutions learn spatial representation, temporal dependence, and spectral dependence. The outputs of the two streams are concatenated for space-time-frequency feature fusion. Extensive experiments implemented on two BCI datasets demonstrate that our model outperforms state-of-the-art MI classification methods.