Reference Bias in Monolingual Machine Translation Evaluation

Marina Fomicheva, Lucia Specia

Abstract: In the translation industry, human translations are assessed by comparison with the source texts. In the Machine Translation (MT) research community, however, it is a common practice to perform quality assessment using a reference translation instead of the source text. In this paper we show that this practice has a serious issue - annotators are strongly biased by the reference translation provided, and this can have a negative impact on the assessment of MT quality.