Analyzing Context Utilization of LLMs in Document-Level Translation
Abstract: Large language models (LLM) are increasingly strong contenders in machine translation. We study document-level translation, where some words cannot be translated without context from outside the sentence. We investigate the ability of prominent LLMs to utilize context by analyzing models' robustness to perturbed (randomized) document context. We find that the strongest translation LLMs are robust to random context in translation performance. However, improved document-translation performance is not always reflected in pronoun translation performance. We highlight the need for context-aware finetuning of LLMs to improve their reliability for document-level translation.
Paper Type: Short
Research Area: Machine Translation
Research Area Keywords: automatic evaluation
Contribution Types: Model analysis & interpretability, NLP engineering experiment
Languages Studied: English, German, French
Submission Number: 3063
Loading