Self-Prompt: Leveraging Gradient-Based Search for Optimizing Prompts in Code Generation

Wenxi Zong, Ge Chen, Yu Tong, Shilong Li, Mou Xiao Feng, ouxuxin, Mingbin Cao

Keywords: Prompt Engineering; Large Language Models; Code Generation

Abstract:

With the rapid advancement of large language models, code generation has witnessed substantial progress. As a crucial factor influencing the quality of generated code, prompt design plays a pivotal role in optimizing model performance. While manual prompt engineering remains a common approach, it is often labor-intensive and suboptimal. To address these limitations, automated prompt optimization techniques have been introduced. However, existing methods that rely on LLMs for automatic prompt construction are inherently constrained by the models' own capabilities, leading to inconsistencies in code generation quality.
In this paper, we propose Self-Prompt, an automated prompt engineering framework tailored for code generation tasks, designed to enhance code quality while ensuring stability and progressive refinement. By leveraging task-specific data, Self-Prompt formulates prompt optimization as a search problem, effectively transforming LLM-based code generation into an iterative prompt refinement process. To assess the effectiveness of Self-Prompt, we conduct extensive experiments using five open-source LLMs across three widely adopted code generation benchmarks: HumanEval, MBPP, and EvalPlus. Employing pass@k as the primary evaluation metric, our results demonstrate that Self-Prompt achieves performance comparable to or exceeding state-of-the-art prompt engineering methods, highlighting its potential for improving automated code generation.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 12440