Hand Graph Representations for Unsupervised Segmentation of Complex Activities

Pratyusha Das, Jiun-Yu Kao, Antonio Ortega, Tomoya Sawada, Hassan Mansour, Anthony Vetro, Akira Minezawa

Abstract: Analysis of hand skeleton data can be used to understand patterns in manipulation and assembly tasks. This paper introduces a graph-based representation of hand skeleton data and proposes a method to perform unsupervised temporal segmentation of a sequence of sub-tasks in order to evaluate the efficiency of an assembly task. We explore the properties of different choices of hand graphs and their spectral decomposition. A comparative performance of these graphs is presented in the context of complex activity segmentation. We show that the spectral graph features extracted from 2D hand motion data outperform the direct use of motion vectors as features. We also make the collected hand position data available to the research community to facilitate further development in this direction.