Explanation for Machine Translation Errors: Generation and Evaluation

Anonymous

Abstract: The fine-grained annotations of translation errors have been widely applied in machine translation researches such as translation quality estimation, designing automatic evaluation metrics, but these annotations only contain information such as error type, location, and severity, the reasons of the errors are not annotated. Since explaining why an annotated text span is erroneous is important for building the trustworthy machine translation models, we manually build the first resource for evaluating the quality of the explanation for the errors. We tested large language models (LLMs) on this evaluation resource, and found that LLMs failed to deliver trustworthy explanations for the machine translation errors. So, we propose a hard chain-of-thought (H-CoT) approach that induces the explanation for the errors step-by-step via hard chains. Experiments on the evaluation resource show that H-CoT greatly improves the explanation quality over LLMs without H-CoT.

Paper Type: long

Research Area: Machine Translation

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Data resources

Languages Studied: English，Chinese，German