Spatial-Temporal Union Channel Enhancement for Continuous Sign Language Recognition

Qingsong Fei, Zhenchao Cui, Guangze Jia

Abstract: The task of sign language recognition (SLR) presents considerable challenges, necessitating a comprehensive grasp of both spatial and temporal visual features for its effective recognition into intelligible language. Nevertheless, existing sign language recognition methods overlook the analysis of different sign language actions, which leads to insufficiently precise and adequate feature extraction. Moreover, existing methods are often too heavy, requiring a lot of computations and hardware resources. They may even require additional facial features and body keypoints for training, and such work undoubtedly increases the training difficulty and computation complexity. To address aforementioned issues, we propose a spatial-temporal union channel enhancement module (STCM) for SLR, which can be flexibly plugged into existing frameworks. The purpose of STCM is to accurately capture and leverage spatial-temporal features and channel-wise features to improving the accuracy of SLR. In order to reduce the computational overhead, we adopt channel partition to group the features. In STCM, we introduce a proficient feature refinement technique that segments the sub-features obtained in the previous step into four groups. Each of these groups is then individually refined using different axial contexts in a parallel manner. Specifically, we develop a suite of efficient element-wise feature enhancers, i.e., UEM/SEM/TEM/CEM. They stand for spatial-temporal union enhancement module, spatial enhancement module, temporal enhancement module and channel enhancement module, respectively. After that, all sub-features are aggregated. Thanks to its multi-perspective attention on sign language action, with little extra computational overhead, STCM achieves new state-of-the-art performance on two publicly available datasets, PHOENIX-2014T and CSL-Daily.