Meaningful Pose-Based Sign Language Evaluation

ACL ARR 2025 May Submission759 Authors

Abstract:

We present a comprehensive study on meaningfully evaluating sign language utterances in the form of human skeletal poses.
The study covers keypoint distance-based, embedding-based, and back-translation-based metrics.
We show tradeoffs between different metrics in different scenarios through automatic meta-evaluation of sign-level retrieval and a human correlation study of text-to-pose translation across different sign languages.
Our findings and the open-source pose-evaluation toolkit provide a practical and reproducible way of developing and evaluating sign language translation or generation systems.

Paper Type: Long

Research Area: Machine Translation

Research Area Keywords: Generation, Machine Translation, Multimodality and Language Grounding to Vision, Robotics and Beyond

Contribution Types: NLP engineering experiment, Reproduction study, Publicly available software and/or pre-trained models, Data resources, Data analysis, Position papers, Surveys

Languages Studied: American Sign Language, Swiss-German Sign Language, French Sign Language, Italian Sign Language

Submission Number: 759