Diagnosis of Human–Object Interaction Detectors for Real-World Educational Applications

Divya Mereddy, Ashwin T S, Marcos Quinones Grueiro, Gautam Biswas

Keywords: Human Object Interaction Models, Education Data, HOI Models Diagnosis

TL;DR: Diagnosis of Human–Object Interaction Detectors for Real-World Educational Applications

Abstract: Human–object interaction (HOI) recognition is critical for automatically analyzing student behavior in complex educational environments. Although state-of-the-art (SOTA) HOI detectors perform well on benchmark datasets, their performance often degrades when deployed in real-world training environments due to domain-specific objects, occlusions, and complex visual conditions. In this paper, we introduce a diagnosis-driven framework that integrates a triplet-level HOI error taxonomy with error–factor attribution analysis for real-world educational video data. We study this problem in the context of Critical Care Air Transport Team (CCATT) mixed-reality medical training. Based on an analysis of HOI failure modes and their causes, we develop a diagnosis-informed refinement strategy for adapting pretrained HOI models to the target domain. Experiments on the CCATT dataset show that this approach improves the macro-F1 score of a pretrained CDN model from 48.6 to 90.2 through targeted refinement guided by diagnosed error factors. These results highlight the importance of detailed diagnostic analysis for enabling reliable deployment of HOI models in real-world educational environments.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Track: Proceeding Track

Submission Number: 29