LLM-based Sign Language Production

Wellington Silveira, Luca Mendonça, Rodrigo de Bem

Abstract: Sign language is an effective means of communication for individuals with different degrees of hearing impairment. Assistive technologies are a significant ally in the social inclusion of people from these groups. Recent advancements in natural language models and high-definition image generation emerged as a novel, non-intrusive approach for tasks related to sign language recognition (SLR), translation (SLT), and production (SLP). This work introduces a novel sign language production approach based on Large Language Models (LLMs) capable of translating text to signs, which are then synthesized into sequences of images. Our method is tested with two LLMs, the LLaMA-7B-hf (quantized) and the Vicuna-7B. Experiments performed on two sign language datasets, the RWTH-PHOENIX-Weather, and the SynLibras-Pose, have shown promising results for this cross-modal application of LLMs.

External IDs:dblp:conf/icmla/SilveiraMB24