Automated Annotation Assistance for Pidgin Sign Japanese in Sign Language Recognition

Gibran Benitez-Garcia, Nobuko Kato, Yuhki Shiraishi, Hiroki Takahashi

Abstract: Japanese Sign Language (JSL) and Manually Coded Japanese (MCJ) are the two main forms of sign language used in Japan. The former differs significantly from spoken Japanese, while the latter closely aligns with spoken syntax. Pidgin Sign Japanese (PSJ) is an intermediate form that heavily relies on nonmanual signals such as facial expressions and head movements for grammatical nuances. Current sign language recognition (SLR) systems predominantly focus on MCJ, neglecting the challenging properties of PSJ. This paper proposes an annotation assistance tool designed to automate the annotation of non-manual and spatial elements in PSJ. Our tool significantly reduces the manual effort required for annotation by using state-of-the-art methods for tracking human pose, hand, and face landmarks, along with recognizing facial action units (FAUs). Validation on a preliminary dataset of 30 videos containing over 90 instances of nonmanual elements demonstrated a $\mathbf{4 0 \%}$ reduction in annotation time, highlighting our proposal’s efficiency and effectiveness in handling the complexities of PSJ.

External IDs:dblp:conf/cw/BenitezGarciaKST24