QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translation
Keywords: Machine Translation, Decoding, Quality Estimation
TL;DR: This paper presents a method to generate diverse and high-quality machine translations by sampling from a Gibbs distribution using the Metropolis-Hastings algorithm.
Abstract: An important challenge in machine translation (MT) is to generate high-quality and diverse translations.
Prior work has shown that the estimated likelihood from the MT model correlates poorly with translation quality.
In contrast, quality evaluation metrics (such as COMET or BLEURT) exhibit high correlations with human judgments, which has motivated their use as rerankers (such as quality-aware and minimum Bayes risk decoding). However, relying on a single translation with high estimated quality increases the chances of "gaming the metric''.
In this paper, we address the problem of sampling a set of high-quality and diverse translations.
We provide a simple and effective way to avoid over-reliance on noisy quality estimates by using them as the energy function of a Gibbs distribution. Instead of looking for a mode in the distribution, we generate multiple samples from high-density areas through the Metropolis-Hastings algorithm, a simple Markov chain Monte Carlo approach.
The results show that our proposed method leads to high-quality and diverse outputs across multiple language pairs (English$\leftrightarrow$\{German, Russian\}) with two strong decoder-only LLMs (Alma-7b, Tower-7b).
Primary Area: Natural language processing
Submission Number: 16067
Loading