MAS-LitEval : Multi-Agent System for Literary Translation Quality Assessment

ACL ARR 2025 May Submission4668 Authors

Abstract: Literary translation requires preserving cultural nuances and stylistic elements, which traditional metrics like BLEU and METEOR fail to assess due to their focus on lexical overlap. This oversight neglects the narrative consistency and stylistic fidelity that are crucial for literary works. To address this, we propose MAS-LitEval, a multi-agent system using Large Language Models (LLMs) to evaluate translations based on terminology, narrative, and style. We tested MAS-LitEval on translations of 'The Little Prince' and 'A Connecticut Yankee in King Arthur's Court', generated by various LLMs, and compared it to traditional metrics. MAS-LitEval outperformed these metrics, with top models scoring up to 0.890 in capturing literary nuances. This work introduces a scalable, nuanced framework for Translation Quality Assessment (TQA), offering a practical tool for translators and researchers.

Paper Type: Short

Research Area: Machine Translation

Research Area Keywords: Machine Translation, Evaluation, Language Models, Multilingualism and Cross-Lingual NLP

Contribution Types: NLP engineering experiment

Languages Studied: English, Korean

Submission Number: 4668