Support Point Sets for Improving Contactless Interaction in Geometric Learning for Hand Pose Estimation

Niklas Hermes, Lasse Hansen, Alexander Bigalke, Mattias P. Heinrich

Abstract: Estimation of the hand pose of a surgeon is becoming more and more important in the context of computer assisted surgery. Previous point cloud-based neural network methods for this task usually estimate offset fields to infer the 3D joint positions. Occlusions of important hand parts and inconsistencies in the point clouds, e.g. caused by uneven exposure from the depth sensor, pose a challenge to these methods. We propose to simplify the optimization problem by only estimating a weight for each point of the cloud, such that the inferred joint position is given as the weighted sum over the input points. To better capture the directional information, we define a support point set that expands the convex hull of the hand point set and enter the union of both sets as input to our network. We propose a hierarchical graph CNN, whose graph structure enables optimal information flow between the two point sets. With a mean joint error of 9.43 mm, our approach outperforms most comparable state-of-the-art methods with an average reduction by 19%, while also reducing the computational complexity.