A Cross-subject and Cross-modal Model for Multimodal Emotion Recognition

Jian-Ming Zhang, Xu Yan, Ziyi Li, Li-Ming Zhao, Yu-Zhong Liu, Hua-Liang Li, Bao-Liang Lu

Abstract: The combination of eye movements and electroencephalography (EEG) signals, representing the external subconscious behaviors and internal physiological responses, respectively, has been proved to be a dependable approach with high interpretability. However, EEG is unfeasible to be put into practical applications due to the inconvenience of data acquisition and inter-subject variability. To take advantage of EEG without being restricted by its limitations, we propose a cross-subject and cross-modal (CSCM) model with a specially designed structure called gradient reversal layer to bridge the modality differences and eliminate the subject variation, so that the CSCM model only requires eye movements and avoids using EEG in real applications. We verify our proposed model on two classic public emotion recognition datasets, SEED and SEED-IV. The competitive performance not only illustrates the efficacy of CSCM model but also sheds light on possible solutions to dealing with cross-subject variations and cross-modal differences simultaneously which help make effective emotion recognition practicable.