Assessment of comparative evaluation techniques for signing agents: a study with deaf adults

Alfarabi Imashev, Nurziya Oralbayeva, Gulmira Baizhanova, Anara Sandygulova

Abstract: Sign languages are considered fully-fledged and complete natural languages that are utilized by individuals who are deaf or hard of hearing as a means of communication within the visual-gestural modality. The utilization of virtual avatars as virtual assistants has witnessed a notable surge over the course of the previous fifteen years. Research on sign language recognition has already shown significant potential in achieving reliable and efficient automatic sign language recognition. Nevertheless, the development of physiologically believable (naturally looking) sign language synthesis and generation techniques is currently in its nascent stages. Moreover, traditional models often are rule-based, rely on manually programmed commands, and require the expertise of proficient interpreters, whereas data-driven approaches have the potential to offer more advanced solutions. In addition to the advancement of sign language systems, scholarly investigations indicate a notable lack in the signing systems evaluation by individuals who utilize sign language (deaf signers and interpreters). In this study, we introduce a sign language interpreting avatar based on data-driven techniques. Additionally, we conduct a subjective evaluation of the avatar’s performance. This paper presents the findings of a study conducted with deaf signers, which aimed to compare three different signing agents to a highly skilled sign language human interpreter. The study utilized well-known metrics that are considered to provide valuable insights into participants’ perceptions of signing agents, also their respective advantages and limitations.