Handshape-Aware Sign Language Recognition: Extended Datasets and Exploration of Handshape-Inclusive Methods
Submission Type: Regular Long Paper
Submission Track: Speech and Multimodality
Submission Track 2: Language Grounding to Vision, Robotics and Beyond
Keywords: Sign language recognition, handshape
TL;DR: This paper extends the existing PHOENIX14T dataset with handshape labels and proposes novel handshape-aware sign language recognition methods, which constantly outperform the baseline systems.
Abstract: The majority of existing work on sign language recognition encodes signed videos without explicitly acknowledging the phonological attributes of signs. Given that handshape is a vital parameter in sign languages, we explore the potential of handshape-aware sign language recognition. We augment the PHOENIX14T dataset with gloss-level handshape labels, resulting in the new PHOENIX14T-HS dataset. Two unique methods are proposed for handshape-inclusive sign language recognition: a single-encoder network and a dual-encoder network, complemented by a training strategy that simultaneously optimizes both the CTC loss and frame-level cross-entropy loss. The proposed methodology consistently outperforms the baseline performance. The dataset and code can be accessed at: www.anonymous.com.
Submission Number: 4466
Loading