When natural language is not enough: The limits of in-context learning demonstrations in multilingual reasoning
Abstract: Previous studies have demonstrated the effectiveness of reasoning methods in eliciting multi-step reasoned answers from Large Language Models (LLMs) by leveraging in-context demonstrations. These methods, exemplified by Chain-of-Thought (CoT) and Program-Aided Language Models (PAL), have been shown to perform well in monolingual contexts, primarily in English. There has, however, been limited exploration of their abilities in other languages.
To gain a deeper understanding of the role of reasoning methods for in-context demonstrations, we propose a multidimensional analysis in languages beyond English, focusing on arithmetic and symbolic reasoning tasks. Our findings indicate that the effectiveness of reasoning methods varies significantly across different languages and models. Specifically, CoT, which relies on natural language demonstrations, tends to be more effective in high-resource languages. Conversely, the structured nature of PAL in-context demonstrations facilitates multilingual comprehension, enabling LLMs to generate programmatic answers in both high- and low-resource languages. This leads to significant improvements in the accuracy and quality of the generated responses.
Paper Type: Long
Research Area: Multilingualism and Cross-Lingual NLP
Research Area Keywords: Multilingualism in NLP
Contribution Types: Model analysis & interpretability, NLP engineering experiment, Reproduction study, Approaches to low-resource settings, Approaches low compute settings-efficiency
Languages Studied: English , German , Russian , French , Spanish , Chinese , Vietnamese , Turkish , Arabic , Italian , Thai, Bulgarian, Urdu , Swahili Hindi
Submission Number: 247
Loading