RuSignBot: Russian Sign Language Synthesis via Customized MimicMotion

Daria Bondarenko, Emilia Bojarskaja, Maxim Novopoltsev, Aleksandr Tulenkov, Ruslan Murtazin, Iuliia Zemtsova, Ilya Makarov, Andrey Savchenko

Keywords: Sign Language Generation, sign language, video generation, text-to-video

TL;DR: We present RuSignBot, an AI system that generates realistic sign language videos from text, validated by both automatic metrics (PSNR, SSIM, APE) and proposed human evaluations of sign recognition by fluent signers, and released as a Telegram bot.

Abstract: The hearing-impaired community is often underserved due to barriers such as a lack of linguistically appropriate services, especially in non-native English-speaking countries. This paper presents RuSignBot, a novel sign language generation for Russian language based on an adapted MimicMotion architecture. To enhance the realism and expressiveness of output videos, we introduce a domain-specific fine-tuning strategy on a large-scale sign language corpus. Quantitative evaluation demonstrates that our fine-tuned model achieves superior performance in standard full-reference metrics, specifically SSIM and PSNR, compared to its base version. Furthermore, we propose a human-centric Sign Understandability Score and conduct a user study with fluent signers. The results confirm that the generated signs are recognized with high accuracy, underscoring the model's communicative efficacy. To facilitate practical application, we integrate the model into a Telegram-based application that converts user-input text into animated sign language videos. The system supports both default and user-defined avatars, highlighting its potential for real-world deployment in assistive technology contexts. The source code and pre-trained models will be publicly released to encourage further research.

Submission Number: 31