Variance-Aware Machine Translation Test Sets
Keywords: Machine Translation, Evaluation, Test Sets
TL;DR: We release 70 small and discriminative test sets for machine translation evaluation.
Abstract: We release 70 small and discriminative test sets for machine translation (MT) evaluation called variance-aware test sets (VAT), covering 35 translation directions from WMT16 to WMT20 competitions. VAT is automatically created by a novel variance-aware filtering method that filters the indiscriminative test instances of the current MT benchmark without any human labor. Experimental results show that VAT outperforms the original WMT benchmark in terms of the correlation with human judgment across mainstream language pairs and test sets. Further analysis on the properties of VAT reveals the challenging linguistic features (e.g., translation of low-frequency words and proper nouns) for the competitive MT systems, providing guidance for constructing future MT test sets. The test sets and the code for preparing variance-aware MT test sets are freely available at https://github.com/NLP2CT/Variance-Aware-MT-Test-Sets.
Supplementary Material: zip
URL: https://github.com/NLP2CT/Variance-Aware-MT-Test-Sets
Contribution Process Agreement: Yes
Dataset Url: https://github.com/NLP2CT/Variance-Aware-MT-Test-Sets
License: The variance-aware test sets were created based on the original WMT test set. Thus, we follow the original data licensing plan already stated by WMT organizers, which is that “The data released for the WMT news translation task can be freely used for research purposes, we just ask that you cite the WMT shared task overview paper, and respect any additional citation requirements on the individual data sets. For other uses of the data, you should consult with original owners of the data sets.” (quoted from the “LICENSING OF DATA” part in the WMT official website).
Author Statement: Yes
Community Implementations: [ 5 code implementations](https://www.catalyzex.com/paper/arxiv:2111.04079/code)
14 Replies
Loading