The Impact of Reasoning Methods across Languages
Abstract: Previous works have been showing the effective operation of reasoning methods to elicit Large Language Models (LLMs) in delivering multi-step reasoned answers. Although these methods, best exemplified by Chain-of-Thought (CoT) and Program-Aided Language Models (PAL), have demonstrated actual functionalities in monolingual (English), multi-, and cross-lingual scenarios, are under-explored and lack an in-depth understanding. To address this gap, we propose a multidimensional analysis using five Cross-lingual tasks, experimenting with the impact of reasoning methods in different LLMs selected per families and scope of construction. Our results reveal that the effectiveness of reasoning methods varies significantly across models, tasks, and languages.
In particular, higher-parameter LLMs, when elicited via CoT, are able to deliver reasoned multi-step answers better than smaller LLMs. In contrast, LLMs prompted via PAL achieve significant improvements anyway of the number of parameters. Finally, by analyzing the role of in-context cross-lingual demonstrations, we reveal that although they may provide benefits significantly in low-resource scenarios, their effectiveness is related to a proper trade-off between quantity and quality of demonstrations.
Paper Type: Long
Research Area: Multilingualism and Cross-Lingual NLP
Research Area Keywords: Reasoning Methods, Cross-Lingual NLP
Contribution Types: Model analysis & interpretability, Reproduction study, Approaches to low-resource settings, Approaches low compute settings-efficiency, Data resources, Data analysis
Languages Studied: English , German , Russian , French , Spanish , Chinese , Vietnamese , Turkish , Arabic , Greek , Thai, Bulgarian, Urdu , Swahili Hindi
Submission Number: 170
Loading