Prompt Engineering a Prompt Engineer

Qinyuan Ye, Mohamed Ahmed, Reid Pryzant, Fereshte Khani

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: prompt engineering, large language models, optimization

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

Prompt engineering is a challenging yet crucial task for optimizing the performance of large language models (LLMs). It requires complex reasoning to examine the model's errors, hypothesize what is missing or misleading in the current prompt, and communicate the task clearly to the LLM. While recent works indicate that LLMs can be meta-prompted to perform automatic prompt engineering, their potentials are not fully unlocked as the meta-prompts may not offer sufficient guidance to elicit complex reasoning capabilities in LLMs.
In this work, we investigate the problem of "prompt engineering a prompt engineer"---constructing a meta-prompt that more effectively guides LLMs to perform prompt engineering. We introduce and analyze key components, such as a step-by-step reasoning template and context specification, which leads to improved performance on automatic prompt engineering. The resulting method, named PE2, finds a prompt that outperforms ``let’s think step by step’’ by 6.3\% on the MultiArith dataset and 3.1\% on the GSM8K dataset. To demonstrate its versatility, we apply PE2 to the Instruction Induction benchmark, a suite of counterfactual tasks, and a real-world industrial prompt. In these settings, PE2 achieves strong performance and outperforms prior automatic prompt engineering baselines. Further, we show that PE2 makes meaningful and targeted prompt edits, amends erroneous or incomplete prompts, and presents non-trivial counterfactual reasoning abilities.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6421