SignKD: Multi-modal Hierarchical Knowledge Distillation for Continuous Sign Language Recognition
Keywords: Sign Language Recognition, Multi-modal, Hierarchical Knowledge Distillation
Abstract: Continuous sign language recognition (CSLR) plays a crucial role in promoting inclusivity and facilitating communication within the hearing-impaired community. One of the key challenges in CSLR is accurately capturing the intricate hand movements involved. To address this challenge, we propose a multi-modal framework that first combines video, keypoints, and optical flow modalities to extract more representative features. We investigate various fusion techniques to effectively integrate the information from these modalities.
Furthermore, we introduce a hierarchical knowledge distillation (HKD) framework to alleviate the computational burden associated with extracting keypoints and optical flow information. This framework enables the hierarchical transfer of knowledge from multiple modalities to a single-modal CSLR model, ensuring high performance while reducing computational costs.
To evaluate the effectiveness of our approach, we conduct extensive experiments on three benchmark datasets: Phoenix-2014, Phoenix-2014T, and CSL-Daily. The results demonstrate that our approach achieves state-of-the-art performance in CSLR, both in the single-stream and multi-stream settings.
Primary Area: representation learning for computer vision, audio, language, and other modalities
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 2517
Loading