Real-Time Hand Model Estimation from Depth Images for Wearable Augmented Reality Glasses

Bill Zhou, Alex Yu, Joseph Menke, Allen Y. Yang

Abstract: This work presents a hand model estimation method designed specifically with augmented reality (AR) glasses and 3D AR interface in mind. The proposed work is capable of estimating the 3D positions of all ten finger from a single depth image. By leveraging a low-dimensional hand model and exploiting hand geometries from an ego-centric view, we build a lightweight algorithm that is accurate, environment agnostic, and runs in real time on mobile hardware. One major consideration in our design for AR is that the user's hand is likely to interact with planar surfaces since they serve as ideal "touchscreens". As a result, our method will not fail to detect the hand even when the hand is in physical contact with a surface such as a table, wall, or even another palm. Our experiment shows using the CVAR database that the accuracy with clear background at 98% and with cluttered background at around 85%