DiP Benchmark Tests: Evaluation Benchmarks for Discourse Phenomena in MT

Prathyusha Jwalapuram, Barbara Rychalska, Shafiq Joty, Dominika Basaj

Keywords: machine translation, discourse, evaluation, benchmark, testsets, leaderboard

Abstract: Despite increasing instances of machine translation (MT) systems including extrasentential context information, the evidence for translation quality improvement is sparse, especially for discourse phenomena. Popular metrics like BLEU are not expressive or sensitive enough to capture quality improvements or drops that are minor in size but significant in perception. We introduce the first of their kind MT benchmark testsets that aim to track and hail improvements across four main discourse phenomena: anaphora, lexical consistency, coherence and readability, and discourse connective translation. We also introduce evaluation methods for these tasks, and evaluate several competitive baseline MT systems on the curated datasets. Surprisingly, we find that the complex context-aware models that we test do not improve discourse-related translations consistently across languages and phenomena. Our evaluation benchmark is available as a leaderboard at <dipbenchmark1.github.io>.

One-sentence Summary: We introduce first-of-their-kind discourse benchmark testsets and evaluation procedures aimed at tracking improvement in machine translation quality for phenomena like anaphora, coherence & readability, lexical consistency and discourse connectives.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=GiAn77rCyr