Evaluating Web-trained Facial Expression Recognition in Naturalistic Collaborative Learning

Sifatul Anindho, Videep Venkatesha, Nathaniel Blanchard

Keywords: Facial Expression Recognition, Affective Computing, Collaborative Problem-Solving, Educational Dataset, Epistemic Emotions

TL;DR: We evaluate whether facial expression recognition models trained on web-scale emotion datasets meaningfully align with epistemic emotions reported in collaborative learning contexts.

Abstract:

Facial expression recognition (FER) have been adopted extensively in educational research. These models are usually trained on web-scale datasets optimized for canonical basic emotions. However, in collaborative learning, the affective states of interest are often epistemic (learning-relevant) in nature—such as confusion, curiosity, and frustration—rather than prototypical basic emotions like disgust or anger. Here, we evaluate popular FER systems trained using web-scale datasets on small-group collaborative problem-solving sessions. We annotate epistemic emotions using a retrospective cued-recall approach, where participants watch a video of a collaborative session immediately after the group task and individually self-report at different segments.
We analyze cross-taxonomy alignment between epistemic emotions and predicted basic emotions, cross-model agreement in collaborative learning, and dimensional valence–arousal structure with respect to basic and epistemic emotions. Across models, categorical predictions overwhelmingly collapse to Neutral for all epistemic labels, and valence–arousal outputs fail to distinguish the different epistemic states. Furthermore, low agreement between models indicates some degree of instability in both web-scale and naturalistic collaborative contexts. Our findings suggest that FER systems trained for canonical emotion recognition on web-scale datasets do not directly generalize to the more subtle epistemic emotions experienced during collaboration, which has implications for education researchers deploying these tools in classrooms. Our code is available at https://anonymous.4open.science/r/cvpr-edu-affect-2026-6EF5.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Track: Proceeding Track

Submission Number: 21