AutoMetrics: Approximate Human Judgments with Automatically Generated Evaluators

ICLR 2026 Conference Submission22092 Authors

Keywords: evaluation, LLM-as-a-judge, metrics, human feedback, open-ended tasks, user-centered evaluation, data-efficient evaluation, automatic metric generation, benchmarking

TL;DR: We use LLMs to automatically generate and validate task-specific evaluation criteria (metrics) that correlate well with human judgements, and release a library/framework for automatic metric induction.

Abstract: Evaluating user-facing AI applications remains a central challenge, especially in open-ended domains such as travel planning, clinical note generation, or dialogue. The gold standard is user feedback (e.g., thumbs up/down) or behavioral signals (e.g., retention), but these are often scarce in prototypes and research projects, or too-slow to use for system optimization. We present **AutoMetrics**, a framework for synthesizing evaluation metrics under low-data constraints. AutoMetrics combines retrieval from **MetricBank**, a collection of 48 metrics we curate, with automatically generated LLM-as-a-Judge criteria informed by lightweight human feedback. These metrics are composed via regression to maximize correlation with human signal. AutoMetrics takes you from expensive measures to interpretable automatic metrics. Across 5 diverse tasks, AutoMetrics improves Kendall correlation with human ratings by up to 33.4% over LLM-as-a-Judge while requiring fewer than 100 feedback points. We show that AutoMetrics can be used as a proxy reward to equal effect as a verifiable reward. We release the full AutoMetrics toolkit and MetricBank to accelerate adaptive evaluation of LLM applications.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 22092