Chinese sign language recognition based on trajectory and hand shape features

Jun He, Zhandong Liu, Jihai Zhang

Abstract: Sign language recognition(SLR) is a challenging task due to the diversity of the signs. To tackle the problem, this paper utilize both trajectory features and hand shape features. Since the trajectory features and hand shape features are not in the same domain, it is unreasonable to concatenate them naively or model them with a unified model. To deal with the issue, we adopt Support Vector Machine(SVM) and validation Hidden Markov Models(VHMM), respectively. To depict the direction of the trajectory, we first employ histogram of oriented displacement(HOD) with SVM to SLR. We propose the relative distance features(RDF) by using VHMM to consider the relationship between hands and the other body parts. As for hand shape feature, we explore histogram of oriented gradient(HOG) in local hand regions with VHMM, too. To facilitate late fusion, we normalize the probabilities of different features to the same range and fuse them for the final classification. To demonstrate the effectiveness of our proposed method, we conduct the experiments both in ChaLearn dataset and our self-build Kinect-based Chinese sign language dataset. The results show that our method outperforms the classical methods and some state-of-the-art methods.