Beyond the Singular: Revealing the Value of Multiple Generations in Benchmark Evaluation

Wenbo Zhang, Hengrui Cai, Wenyu Chen

Keywords: Benchmark Evaluation, Statistical Model, Variance Reduction, Prompt Difficulty

TL;DR: This paper proposes a hierarchical model to account for LLM randomness, reducing variance of the benchmark score estimation and introducing a prompt-level difficulty metric.

Abstract: Large language models (LLMs) have demonstrated significant utility in real-world applications, exhibiting impressive capabilities in natural language processing and understanding. Benchmark evaluations are crucial for assessing the capabilities of LLMs as they can provide a comprehensive assessment of their strengths and weaknesses. However, current evaluation methods often overlook the inherent randomness of LLMs by employing deterministic generation strategies or relying on a single random sample, resulting in unaccounted sampling variance and unreliable benchmark score estimates. In this paper, we propose a hierarchical statistical model that provides a more comprehensive representation of the benchmarking process by incorporating both benchmark characteristics and LLM randomness. We show that leveraging multiple generations improves the accuracy of estimating the benchmark score and reduces variance. Multiple generations also allow us to define $\mathbb P\left(\text{correct}\right)$, a prompt-level difficulty score based on correct ratios, providing fine-grained insights into individual prompts. Additionally, we create a data map that visualizes difficulty and semantics of prompts, enabling error detection and quality control in benchmark construction.

Submission Number: 87