ELPO: Ensemble Learning Based Prompt Optimization for Large Language Models

Qing Zhang, Bing Xu, Xudong Zhang, Yifan Shi, Yang Li, Chen Zhang, Yik Chung WU, Ngai Wong, YijieChen, Hong Dai, XianSen Chen, Mian Zhang

Keywords: automatic prompt optimization, ensemble learning, large language models, prompt engineering

TL;DR: This paper proposes an ensemble learning based algorithm called ELPO to realize robust and accurate results in automatic prompt optimization.

Abstract:

The remarkable performance of Large Language Models (LLMs) highly relies on crafted prompts.
However, manual prompt engineering is a laborious process, creating a core bottleneck for practical application of LLMs.
This phenomenon has led to the emergence of a new research area known as Automatic Prompt Optimization (APO), which develops rapidly in recent years.
Existing APO methods such as those based on evolutionary algorithms or trial-and-error approaches realize an efficient and accurate prompt optimization to some extent.
However, those researches focus on a single model or algorithm for the generation strategy and optimization process, which limits their performance when handling complex tasks.
To address this, we propose a novel framework called $\textbf{E}$nsemble $\textbf{L}$earning based $\textbf{P}$rompt $\textbf{O}$ptimization (ELPO) to achieve more accurate and robust results.
Motivated by the idea of ensemble learning, ELPO conducts voting mechanism and introduces shared generation strategies along with different search methods for searching superior prompts.
Moreover, ELPO creatively presents more efficient algorithms for the prompt generation and search process.
Experimental results demonstrate that ELPO outperforms state-of-the-art prompt optimization methods across different tasks, e.g., improving F1 score by 7.6 on ArSarcasm dataset.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 7573