Emotional Affect Estimation Using Video and EEG Data in Deep Neural Networks

Arvid Frydenlund, Frank Rudzicz

Abstract: We present a multimodel system for independent affect recognition using deep neural networks. Using the DEAP data set, features are extracted from EEG and other physiological signals, as well as videos of participant faces. We introduce both a novel way of extracting video features using sum-product networks, and a unique method of creating extra training examples from data that would have otherwise been lost in downsampling. Deep neural networks are used for estimating the emotional dimensions of arousal, valence, and dominance, along with favourability and familiarity. This work lays the foundation for future work in estimating emotional responses from physiological measurements.