Annotating Student Engagement Across Grades 1-12: Associations with Demographics and Expressivity

Nese Alyüz, Sinem Aslan, Sidney K. D'Mello, Lama Nachman, Asli Arslan Esme

Abstract: Digital learning technologies that aim to measure and sustain student engagement typically use supervised machine learning approaches for engagement detection, which requires reliable “ground-truth” engagement annotations. The present study examined associations between student demographics (age [grade], gender, and ethnicity) and the reliability of engagement annotations based on visual behaviors. We collected videos of diverse students (N = 60) from grades 1–12 who engaged in one-hour online learning sessions with grade-appropriate content. Each student’s data was annotated by three trained coders for behavioral and emotional engagement. We found that inter-rater reliability (IRR) for behavioral engagement was higher for older students whereas IRRs for emotional engagement was higher for younger students. We also found that both rotational head movements and facial expressivity decreased with age, and critically, rotational head movements mediated the effects of grade on behavioral IRR; there was no mediation for emotional IRR. There were no effects of gender or ethnicity on IRR. We discuss the implications of our findings for annotating engagement in supervised learning models for diverse students and across grades.

External IDs:dblp:conf/aied/AlyuzADNE21