Extending Automatic Machine Translation Evaluation to Book-Length Documents

ACL ARR 2025 May Submission673 Authors

Abstract: Despite Large Language Models (LLMs) demonstrating superior translation performance and long-context capabilities, evaluation methodologies remain constrained to sentence-level assessment due to dataset limitations, token number restrictions in metrics, and rigid sentence boundary requirements. We introduce an evaluation scheme that extends existing automatic metrics to long-document translation by treating documents as continuous text and applying sentence segmentation and alignment methods. Our approach enables previously unattainable document-level evaluation, handling translations of arbitrary length generated with document-level prompts while accounting for under-/over-translations and varied sentence boundaries. Experiments show our scheme significantly outperforms existing long-form document evaluation schemes, while comparable to evaluations performed with groundtruth sentence alignments. Additionally, we apply our scheme to book-length texts and newly demonstrate that many open-weight LLMs fail to effectively translate documents at their reported maximum context lengths.

Paper Type: Long

Research Area: Machine Translation

Research Area Keywords: automatic evaluation

Contribution Types: NLP engineering experiment, Reproduction study, Publicly available software and/or pre-trained models

Languages Studied: Chinese, English, German, Japanese, Spanish

Submission Number: 673