CODA: A Novel End-to-End Computational Framework for Sign-to-Sign Translation Enhanced with LLMs

Anonymous

Abstract: The number of different signed languages presents novel challenges in cross-cultural sign language processing. Our work takes a pioneering step into direct sign-to-sign translation across different sign language families. We first conduct a qualitative analysis of linguistic traits, both shared and distinctive, within a parallel corpus of multiple signed pairs of sentences. We then introduce a novel generation framework, CODA, for translating one sign language to another, employing Large Language models as intermediary text recognizers. We compile a dataset for sign-to-sign translation pairs across three signed languages: American Sign Language (ASL), Chinese Sign Language (CSL), and German Sign Language (DGS). We further utilize sign glosses as an intermediate representation to construct a multi-task model that can assist in preserving the semantic meaning of generated sign skeletal videos. We show that our model performs well on automatic metrics for sign-to-sign translation and generation as a novel first implementation. We make all our code and models available upon acceptance.

Paper Type: long

Research Area: Multilinguality and Language Diversity

Contribution Types: NLP engineering experiment, Publicly available software and/or pre-trained models, Data resources, Data analysis

Languages Studied: German Sign Language, Chinese Sign Language, American Sign Language