Hand-object Interaction based Semi-automatic Objects Annotation for Human Activity Datasets

Yuankai Wu, Zhouyi Gu, Marsil Zakour, Rahul Gopal Chaudhari

Abstract: Objects annotation is a prerequisite for learning-based detectors to locate and extract object semantic information from the scene. This provides a significant contribution to human activity recognition (HAR), which is a core study in computer vision and assistive robotics area. The current dominant HAR approaches focus on the end-to-end model using RGB images or optical flow stream as input rather than using semantic human-object interaction information. This is mainly due to the lacking of object annotations in the current HAR datasets. To this end, we propose a novel hand-object interaction based approach to semi-automatic annotating objects in videos. The proposed approach annotates the objects by mapping the trajectories of the hands to the trajectories of the object bounding boxes, and smoothen the track with the Kalman filter. Unlike existing methods, our approach requires only a few clicks for annotations without fine-tuning any object detection models after annotating objects manually. Our experimental results on the Bimanual Actions Dataset achieve 78.81% accuracy compared with the ground truth objects annotation at an intersection over union (IoU) of 0.5. This outperforms the state-of-the-art tracker by 7.15%, and reduces manual annotation workload by 92.86%.