Estimation of wakefulness in video-based lectures based on multimodal data fusion

Ryosuke Kawamura, Shizuka Shirai, Mehrasa Alizadeh, Noriko Takemura, Hajime Nagahara

Abstract: In distance learning contexts, drowsiness is a major factor which disturbs learning. However, it is not easy for instructors to monitor students' wakefulness. In order to improve learning efficacy, accurate estimation of wakefulness is needed. In this study, we propose a multimodal wakefulness estimation method based on face and body movement information. We utilize web-cameras to obtain facial and head (face-head) movements and pressure mats for body movements, the latter of which can record the distribution of upper body pressure while watching video lectures. To confirm the effectiveness of multimodal data for wakefulness estimation, we conducted an experiment to collect data from students as they engaged in e-learning and their level of wakefulness was annotated in one-second windows. We extracted 45 features from face-head movements, and 80 features from seat pressure data. Two types of fusion methods, early and decision level fusion were applied, and the late fusion approach achieved an average F1-macro score of 0.70 in three levels of wakefulness estimation, which is higher than the unimodal approach. This result indicates that fusion of facial images and seat pressure features can be effective for learner wakefulness estimation.