“Be My Cheese?”: Cultural Nuance Benchmarking for Machine Translation in Multilingual LLMs

ACL ARR 2026 January Submission3897 Authors

Keywords: multilingual, large language models, natural language processing, machine translation

Abstract:

We present a large-scale human evaluation benchmark for assessing cultural localisation in machine translation produced by state-of-the-art multilingual large language models (LLMs). Existing MT benchmarks emphasise token-level and grammatical accuracy, but often overlook pragmatic and culturally grounded competencies required for real-world localisation. Building on a pilot study of 87 translations across 20 languages, we evaluate 7 multilingual LLMs across 15 target languages with 5 native-speaker raters per language. Raters scored both full-text translations and segment-level instances of culturally nuanced language (idioms, puns, holidays, and culturally embedded concepts) on an ordinal 0–3 quality scale; segment ratings additionally included an NA option for untranslated segments.

Across full-text evaluations, mean overall quality is modest (1.68/3): GPT-5 (2.10/3), Claude Sonnet 3.7 (1.97/3), and Mistral Medium 3.1 (1.84/3) form the strongest tier with fewer catastrophic failures. Segment-level results show sharp category effects: holidays (2.20/3) and cultural concepts (2.19/3) translate substantially better than idioms (1.65/3) and puns (1.45/3), and idioms are most likely to be left untranslated. These findings demonstrate a persistent gap between grammatical adequacy and cultural resonance. To our knowledge, this is the first multilingual, human-annotated benchmark focused explicitly on cultural nuance in translation and localisation, highlighting the need for culturally informed training data, improved cross-lingual pragmatics, and evaluation paradigms that better reflect real-world communicative competence.

Paper Type: Long

Research Area: Machine Translation

Research Area Keywords: machine translation, resources and evaluation, multilingualism and cross-lingual NLP, human-centred NLP, discourse and pragmatics

Contribution Types: Data resources, Data analysis

Languages Studied: Afrikaans (South Africa), Arabic (Egypt), Brazilian Portuguese, Cantonese (Hong Kong), Czech, Dutch, Hebrew, Hindi, Japanese, Korean, Mandarin (Taiwan), Russian (Kazakhstan), Spanish (Mexico), Swahili (Kenya), Urdu (Pakistan)

Submission Number: 3897