A multi-scale descriptor for real time RGB-D hand gesture recognition

Yao Huang, Jianyu Yang

Abstract: Highlights • A new RGB-D shape-based hand gesture recognition method is proposed. • A new hand shape descriptor is proposed with emphasized finger feature. • Different recognition engines are explored and compared for different applications. • Sota accuracies are achieved on several benchmarks, as well as excellent efficiency. • A demo video is given for a real life application of human-computer interaction. Abstract The development of depth cameras, e.g., the Kinect sensor, provides new opportunities for human computer interaction (HCI). Although the Kinect sensor has been extensively applied for human tracking, human action recognition and hand gesture recognition, real time hand gesture recognition is still a challenging problem. In this paper, a new real time hand gesture recognition method is proposed. Since fingers are the most important clue for hand gesture classification, a finger-emphasized multi-scale descriptor is proposed. The proposed descriptor incorporates three types of parameters of multiple scales to make a discriminative representation of the hand shape. Furthermore, the features of fingers are emphasized for hand gesture analysis. Three solutions to hand gesture recognition are then investigated with DTW, SVM, and neural network. Extensive experiments are conducted and the results show that the proposed method is robust to noise, articulations and rigid transformations. The comparison with state-of-the-art methods verifies the accuracy and efficiency of our method.