GLaPE: Gold Label-agnostic Prompt Evaluation for Large Language Models
Abstract: Despite the rapid progress of large language models (LLMs), their task performance remains sensitive to prompt design.
Recent studies have explored leveraging the LLM itself as an optimizer to identify optimal prompts that maximize task accuracy.
However, when evaluating prompts, such approaches heavily rely on elusive manually annotated gold labels to calculate task accuracy for each candidate prompt, which hinders its generality.
To overcome the limitation, this work proposes GLaPE, a gold label-agnostic prompt evaluation method to alleviate dependence on gold labels.
GLaPE is composed of two critical aspects: self-consistency evaluation of a single prompt and mutual-consistency refinement across multiple prompts.
Experimental results on 8 widely-recognized reasoning tasks demonstrate that GLaPE can produce more effective prompts, achieving performance comparable to those derived from manually annotated gold labels.
Analysis shows that GLaPE provides reliable evaluations aligned with accuracy, even in the absence of gold labels.
Code is publicly available at **Anonymous**.
Paper Type: Long
Research Area: Resources and Evaluation
Research Area Keywords: evaluation methodologies; metrics
Contribution Types: Approaches to low-resource settings
Languages Studied: English
Submission Number: 3827
Loading