A Deep CNN-BiGRU Network for Multi-stream Hand Gesture Recognition Framework

Nahla Majdoub Bhiri, Safa Ameur, Imen Jegham, Ihsen Alouani, Anouar Ben Khalifa

Abstract: Hand Gesture Recognition (HGR) achieved significant progress through diverse fields due to recent advancements in machine learning and sensor technologies. While Leap Motion Controller sensors offer convenient hand tracking and multi-modal data (skeletal and depth), the heterogeneous nature of these data modalities poses several challenges for HGR systems. In order to exploit the complementary information offered by skeleton and depth data, fusion algorithms are widely used. This paper proposes a novel Deep CNN-BiGRU model incorporating both intermediate and late fusion strategies. For each modality, we use a separate model for feature extraction step. Then, we apply fusion techniques for the decision step. Our proposed model demonstrates superior performance compared with models employed separately on skeletal or depth data, highlighting its effectiveness in exploiting the combined information for robust and accurate HGR.