EEG Emotion Recognition Supervised by Temporal Features of Video Stimuli

Shuang Ran, Wei Zhong, Danting Duan, Fei Hu, Long Ye, Qin Zhang

Abstract: Different types of video stimuli can activate different reactions in the human brain and these signals can be captured and analyzed for emotion recognition applications. However, accurate recognition across subjects is still challenging due to non-stationary and low signal-to-noise ratio of EEG signals. Lying at the intersection of video content analysis, we make an attempt to supervise EEG features with the external evoked video features. An end-to-end framework is proposed by extracting useful emotional representations in EEG signals with the complementarity of video stimuli. For the feature obtaining, an EEG feature extractor and a video feature extractor are combined, along with a cross-modal transformer to align the distributions of the two type features, and then a self-attention mechanism is designed for fusion. The experiments on the subset of DEAP and self-collected datasets validate that the enhancement of EEG features supervised by stimulus information is a reliable solution for subject-independent emotion recognition.