Automated Capability Evaluation of Foundation Models

ICLR 2026 Conference Submission14093 Authors

Keywords: capability evaluation, foundation model evaluation

TL;DR: adaptive and automated capability evaluation of foundation models without exhaustive evaluation

Abstract:

Current evaluation frameworks for foundation models rely heavily on static, manually curated benchmarks, limiting their ability to capture the full breadth of model capabilities. This paper introduces Active learning for Capability Evaluation (ACE), a novel framework for scalable, automated, and fine-grained evaluation of foundation models. ACE leverages the knowledge embedded in powerful frontier models to decompose a domain into semantically meaningful capabilities and generates diverse evaluation tasks, significantly reducing human effort. In Mathematics, ACE generated 433 capabilities and 11,800 tasks, covering 94\% of Wikipedia-defined skills in the domain while introducing novel, coherent ones.
To maximize efficiency, ACE fits a \emph{capability model} in latent semantic space, allowing reliable approximation of a subject model’s performance by evaluating only a subset of capabilities via active learning. It reaches within 0.01 RMSE of exhaustive evaluation by evaluating less than half of capabilities.
Compared to static datasets, ACE provides more balanced coverage and uncovers fine-grained differences that aggregate metrics fail to capture. Our results demonstrate that ACE provides a more complete and informative picture of model capabilities, which is essential for safe and well-informed deployment of foundation models.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 14093