Exploring Context-Aware Evaluation Metrics for Machine Translation
Submission Type: Regular Short Paper
Submission Track: Machine Translation
Submission Track 2: Resources and Evaluation
Keywords: Machine Translation, Automatic Evaluation Metric, Context Awareness
TL;DR: We introduce context-aware methods for MT evaluation and propose our improved Cont-COMET metric.
Abstract: Previous studies on machine translation evaluation mostly focused on the quality of individual sentences, while overlooking the important role of contextual information. Although WMT Metrics Shared Tasks have introduced context content into the human annotations of translation evaluation since 2019, the relevant metrics and methods still did not take advantage of the corresponding context. In this paper, we propose a context-aware machine translation evaluation metric called Cont-COMET, built upon the effective COMET framework. Our approach simultaneously considers the preceding and subsequent contexts of the sentence to be evaluated and trains our metric to be aligned with the setting during human annotation. We also introduce a content selection method to extract and utilize the most relevant information. The experiments and evaluation of Cont-COMET on the official test framework from WMT show improvements in both system-level and segment-level assessments.
Submission Number: 5603
Loading