Affective annotation of videos from EEG-based crowdsourcing

Yoelvis Moreno-Alcayde, Tuukka Ruotsalo, Luis A. Leiva, V. Javier Traver

Abstract: Enriching multimedia content with affective metadata unlocks opportunities for innovative and enhanced user experiences in recommendation systems, multimedia retrieval, and many other applications. However, it is really challenging to accurately decode human affect. On the one hand, manual procedures for affective annotation are labour-intensive and hardly scalable. On the other hand, content-based approaches rely on the assumption that affective experiences (1) are directly related to the content, (2) are homogeneous for a particular content, and (3) ignore the subjective nature of affective responses. Recent advancements in brain-computer interfacing (BCI) signifies the prospect of partially automating affective annotation by decoding natural affective experiences toward the content. We consider affective annotation of videos based on brainsourcing: crowdsourced affective reactions from brain signals, recorded while participants were watching videos. Our experiments are based on two popular datasets (DEAP and SEED) and three crowdsourcing models. Crowdsourcing models can support affective annotation for all videos in the SEED dataset and most videos in the DEAP dataset. For both datasets and crowdsourcing models, the performance of affective annotation increases with the crowd size and shows increased confidence of the classifiers with larger crowd sizes. For example, the average classification accuracy for binary valence in DEAP is less than 60% for the individual predictions, but increases up to about 70% for a crowd size of six participants and get to about 80% for fourteen participants. Our findings open avenues for utilizing data captured via BCI for understanding and annotating content according to its users’ affective experiences.

External IDs:dblp:journals/paa/MorenoAlcaydeRLT25