Automatic and Interactive Annotation of Non-manual and Spatial Features in Pidgin Sign Japanese for SLR

Gibran Benitez-Garcia, Nobuko Kato, Yuhki Shiraishi, Hiroki Takahashi

Abstract: Pidgin Sign Japanese (PSJ), an intermediate form between Japanese Sign Language (JSL) and Manually Coded Japanese (MCJ), presents unique challenges for annotation due to its reliance on non-manual elements such as head movements and facial expressions. Building upon our previous work on automating the annotation of these elements, we present an improved annotation tool that enhances both detection accuracy and annotation efficiency. The tool significantly reduces manual effort by leveraging state-of-the-art methods for tracking human pose, hand, and face landmarks, along with recognizing facial action units (FAUs). Furthermore, it introduces interactive refinement capabilities to address common issues such as missing or inaccurate hand keypoints, including copy-pasting across frames and interpolation of hand positions. Evaluations on a preliminary PSJ dataset demonstrate a reduction in annotation time and improved usability compared to previous tools, supporting the creation of high-quality PSJ corpora for future Sign Language Recognition (SLR) research.

External IDs:dblp:conf/acivs/BenitezGarciaKST25