Multimodal Distillation for Video-Based Sleep Behavior Analysis

Jacopo Donà, Marco Carletti, Davide Tonon, Gianluca Rossato, Vittorio Murino

Abstract: Sleep is essential for human wellbeing and good health. One of the diseases impacting sleep quality is the so-called REM (Rapid Eye Movements) sleep behavior disorder (RBD), which can be due to serious neuro-degenerative pathologies. Typically, sleep is investigated by monitoring the patient during sleep adopting the polysomnography (PSG), which is the standard examination protocol for diagnosing and monitoring RBD events. Unfortunately, PSG is a costly and invasive procedure, requiring the patient to be physically connected to multiple sensors recording physiological signals while being monitored by a video camera. In our study, we demonstrate how multimodal distillation can be leveraged to perform RBD event classification based on video data only. By casting the problem as a binary classification task, a teacher model distills multimodal PSG physiological data to guide a student model that relies solely on the input video stream. Our method improves F1-score by up to 7.5% over the baseline, demonstrating the feasibility of non-invasive, cost-effective video-based RBD classification systems.

External IDs:doi:10.1007/978-3-032-10192-1_51