MultiPoT: Multilingual Program of Thoughts Harnesses Multiple Programming Languages

Anonymous

Abstract: Program of Thoughts (PoT) is an approach characterized by its executable intermediate steps, which ensure the accuracy of the numerical calculations in the reasoning process. Currently, PoT primarily uses Python. However, relying solely on a single language may result in suboptimal solutions and overlook the potential benefits of other programming languages. In this paper, we conduct comprehensive experiments on the programming languages used in PoT and find that no single language consistently delivers optimal performance across all tasks and models. The effectiveness of each language varies depending on the specific scenarios. Inspired by this, we propose a task and model agnostic approach called MultiPoT, which harnesses strength and diversity from various languages. Experimental results reveal that it significantly outperforms Python Self-Consistency. Furthermore, it achieves comparable or superior performance compared to the best monolingual PoT in almost all tasks across all models. In particular, MultiPoT achieves more than 4.6\% improvement on average on both Starcoder and ChatGPT(gpt-3.5-turbo).

Paper Type: long

Research Area: NLP Applications

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English,Python,R,C++,Java,JavaScript

Preprint Status: We are considering releasing a non-anonymous preprint in the next two months (i.e., during the reviewing process).

A1: yes

A1 Elaboration For Yes Or No: Section 8

A2: no

A2 Elaboration For Yes Or No: The data and models we used are publicly available and no risk.

A3: yes

A3 Elaboration For Yes Or No: Section 1,7

B: yes

B1: yes

B1 Elaboration For Yes Or No: Section 4

B2: no

B2 Elaboration For Yes Or No: The data and models we used are publicly available and no risk.

B3: n/a

B4: n/a

B5: n/a

B6: yes

B6 Elaboration For Yes Or No: Section 9

C: yes

C1: yes

C1 Elaboration For Yes Or No: Section 4

C2: yes

C2 Elaboration For Yes Or No: Section 4

C3: yes

C3 Elaboration For Yes Or No: Section 5

C4: no

C4 Elaboration For Yes Or No: I only use accuarcy.

D: no

D1: n/a

D2: n/a

D3: n/a

D4: n/a

D5: n/a

E: no

E1: n/a