Attention-Based Multi-layer Perceptron to Categorize Affective Videos from Viewer's Physiological Signals

Lazib Sharar Shaiok, Ishtiaqul Hoque, Md. Rakibul Hasan, Shreya Ghosh, Tom Gedeon, Md. Zakir Hossain

Abstract: The rapid growth of online video content has led to an increasing demand for effective video categorization methods. Current methods employed by video platforms include ratings from moderators, creators, and viewers. However, such a self-rating categorization method might not be the most efficient or insightful way to categorize videos. If physiological signals were taken into account, that would make the categorization more robust and could provide content creators, advertisers, and researchers with a better understanding of the viewers’ emotional responses and preferences. In this paper, we develop a hybrid MLP architecture called “ATT-MLP” that utilizes self-attention in its layers and then test its performance on the AVDOS (Affective Video Dataset Online Study) dataset – a database where viewers’ physiological signals were measured whilst they watched pre-classified videos. ATT-MLP outperformed MLP and traditional ML algorithms (Gaussian Support Vector Machine (SVM), K-Nearest Neighbors (KNN), Linear Ridge, and Random Forrest) across all five data modalities (HRV, IMU, EMG-A, EMG-C, and ALL) of the AVDOS dataset. Accuracy and F1 were used as performance metrics, and the hybrid MLP architecture recorded the highest accuracy and F1 score, 93.8% and 93.1%, when the EMG-A data modality of the AVDOS dataset was used. This study shows that the MLP employing self-attention mechanisms within its hidden layers can be a powerful tool in the classification tasks of affective datasets. The code for the aforementioned model is publicly available on Github: https://github.com/IshtiaqHoque/ATT-MLP.