Accurate Hand Gesture Recognition Using Color and Depth Images with Modality-invariant Fusion

Masaya Okano, Jiaqing Liu, Tomoko Tateyama, Yutaro Iwamoto, Yen-Wei Chen

Abstract: Gesture recognition is actively used, and has been applied in various fields, including games and medicine. For accurate gesture recognition, multi-modal information with color and depth images has recently been used. In multimodal gesture recognition, the fusion of color and depth images is crucial. To date, early and late fusion approaches have been widely used for the fusion of color and depth images. However, the enhancement of performance is limited due to the gap between the modalities. In this study, we proposed a modality-invariant fusion approach to overcome the modality gap issue. We applied the proposed approach to a public and our private data set and verified its effectiveness.