Do Multilingual Language Models Think Better in English?

Anonymous

TL;DR: We show that using an LLM to translate its input into English and performing the task over the translated input works better than using the original non-English input.

Abstract: Translate-test is a popular technique to improve the performance of multilingual language models. This approach works by translating the input into English using an external machine translation system before running inference. However, these improvements can be attributed to the use of a separate translation system, which is typically trained on large amounts of parallel data not seen by the language model. In this work, we introduce a new approach called self-translate that leverages the few-shot translation capabilities of multilingual language models. This allows us to analyze the effect of translation in isolation. Experiments over 5 tasks show that self-translate consistently outperforms direct inference, demonstrating that language models are unable to leverage their full multilingual potential when prompted in non-English languages.

Paper Type: short

Research Area: Multilinguality and Language Diversity

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Approaches to low-resource settings

Languages Studied: ar, bg, bn, de, el, en, es, et, eu, fr, hi, ht, id, it, ja, ko, my, qu, ru, sw, ta, te, th, tr, ur, vi, zh