$\mathrm {M^{3}S \text{- }CNN}$: Resting-State EEG Based Multimodal and Multiscale Feature Extraction for Student Behavior Prediction in Class

Yehan Xu, Jingyu Luo, Junyi Liang, Shaogang Song, Ming Ma, Ziheng Guo, Likun Xia

Abstract: Intelligent video monitoring and analysis enable correction of personalized learning behavior from a quantitative perspective. Unfortunately, such approaches can only suggest individual concentration level/status and thinking activity based on their appearance, which is subjective. Although resting-state electroencephalogram (EEG) has been recognized as an indicator which is closely related to people’s thinking activities, EEG based models have not yet provided sufficient solutions due to following reasons: (1) insufficient extraction of features due to single modality of input signal; (2) lack of attention to multiscale features caused by the convolutions with single kernel size. To address the issues above, we propose the following solutions. Firstly, a paradigm is designed for extracting resting-state EEG before class (pre-class) and after class (post-class) instead of video monitoring in class. Secondly, we propose a novel framework named multimodal and multiscale Convolutional Neural Network (\(\mathrm { M^{3}S \text{- }CNN}\)) for feature extraction, which consists of two modules: (1) a feature extraction module with Gramian Angular Summation Field (GASF) and Gramian Angular Difference Field (GADF), aiming to transform the EEG features into forms of pictures and (2) a multiscale module, aiming to improve feature extraction ability by grouping different convolution kernel sizes. Finally, a number of classifiers are employed for student status identification. \(\mathrm{M^{3}S\text{- }CNN}\) is evaluated using one private dataset for three classes, which is divided into training, validation and test sets using an 8:1:1 ratio. Experimental results demonstrate that \(\mathrm{M^{3}S\text{- }CNN}\) along with the classifier of Random Forest (RF) is superior to others with accuracy of 99.77%. This indicates the viability of the proposed model for identification of student status during class.