Code prompting: A Systematic Study of How to Improve Program-based Prompting for Large Language Model Reasoning

Anonymous

TL;DR: We perform a systematic study of how to improve program-based prompting for LLM reasoning.

Abstract:

Large language models (LLMs) have scaled up to unlock a wide range of complex reasoning tasks with the aid of various prompting methods. However, previous prompting methods generate natural language intermediate steps to help reasoning, which can cause imperfect task reduction and confusion due to the ambiguity and sequential nature of natural language. To mitigate such limitations, [1,2] have proposed program-based prompting, triggering code as intermediate steps. In this paper, we perform a systematic study of the approach which we refer to as "code prompting". We conduct experiments on both symbolic and arithmetic reasoning datasets regarding both zero-shot/few-shot scenarios, whether to employ an external interpreter for code execution or use the LLM itself instead, and auxiliary prompting techniques to facilitate reasoning including "self-debugging", "comments", "equation instruction" and "elimination of irrelevant information". To further understand the performance and limitations of code prompting, we perform extensive ablation studies and error analyses. We also consider the ensemble of code prompting and CoT prompting to combine the strengths of both.

[1] Gao et al, 2023. Pal: Program-aided language models.
[2] Chen et al, 2022. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.

Paper Type: long

Research Area: Question Answering

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Publicly available software and/or pre-trained models, Data analysis

Languages Studied: English