Structured Chain-of-Thought Prompting Enhances Code Generation with Large Language Models

Anonymous

Abstract:

Chain-of-Thought (CoT) prompting with Large Language Models (LLMs) has shown impressive abilities in code generation. However, the accuracy of Chain-of-Thought prompting still can not satisfy practical applications. For example, gpt-3.5-turbo with Chain-of-Thought prompting only achieves 53.29% Pass@1 in HumanEval.
This paper proposes the Structured Chain-of-Thought (SCoT) and presents SCoT prompting. Our motivation is source code contains rich structural information. Intuitively, structured intermediate reasoning steps make for structured source code.
SCoT prompting teaches LLMs to generate a SCoT and then output the code. A SCoT is a series of intermediate reasoning steps built with program structures.
By explicitly generating program structures, LLMs' programming abilities are further unlocked, i.e., learning to think about how to solve requirements using the programming logic.
We apply SCoT prompting to two LLMs (i.e., OpenAI gpt-3.5-turbo and code-davinci-002) and evaluate it on three benchmarks (i.e., HumanEval, MBPP, and MBCPP). SCoT prompting outperforms Chain-of-Thought prompting by up to 13.79% in Pass@1. SCoT prompting is robust to examples and achieves substantial improvements. The human evaluation also shows human developers prefer programs from SCoT prompting.

Paper Type: long

Research Area: NLP Applications

Languages Studied: Programming Languages