Does GPT-3 Produces Less Literal Translations?

Anonymous

Abstract: Large Language Models (LLMs) such as GPT-3 have emerged as general purpose language models capable of addressing any natural language generation or understanding task. On the task of Machine Translation (MT), multiple works have investigated few-shot prompting mechanisms to elicit better translations from LLMs. However, there has been relatively little investigation on how such translations qualitatively differ from the translations generated by Neural Machine Translation (NMT) models. In this work, we focus on translation literalness as a property to better differentiate the characteristics of translations from LLMs in the GPT family. We show that E-X translations from GPT-3, even though achieving similar (or better) quality estimates than NMT models, incur a significantly higher number of unaligned source words as well as higher non-monotonicity, which indicates a bias towards less literal translations. We show that this effect also becomes apparent in human evaluations of translation literalness. We further investigate this hypothesis by conducting experiments on sentences with idioms (both natural as well as synthetic), wherein the desired translations themselves admit greater figurativeness.