Position: AI Evaluation Should Learn from How We Test Humans

Yan Zhuang, Qi Liu, Zachary Pardos, Patrick C. Kyllonen, Jiyun Zu, Zhenya Huang, Shijin Wang, Enhong Chen

TL;DR: This position paper argues that psychometrics, a theory originating in the 20th century for human assessment, could be a powerful solution to the challenges in today's AI evaluations

Abstract: As AI systems continue to evolve, their rigorous evaluation becomes crucial for their development and deployment. Researchers have constructed various large-scale benchmarks to determine their capabilities, typically against a gold-standard test set and report metrics averaged across all items. However, this static evaluation paradigm increasingly shows its limitations, including high evaluation costs, data contamination, and the impact of low-quality or erroneous items on evaluation reliability and efficiency. In this Position, drawing from human psychometrics, we discuss a paradigm shift from static evaluation methods to adaptive testing. This involves estimating the characteristics or value of each test item in the benchmark, and tailoring each model's evaluation instead of relying on a fixed test set. This paradigm provides robust ability estimation, uncovering the latent traits underlying a model’s observed scores. This position paper analyze the current possibilities, prospects, and reasons for adopting psychometrics in AI evaluation. We argue that *psychometrics, a theory originating in the 20th century for human assessment, could be a powerful solution to the challenges in today's AI evaluations*.

Lay Summary: As AI systems, especially LLMs, continue to advance, rigorous and trustworthy evaluation becomes increasingly critical for their development and deployment. Today, AI is typically assessed using large collections of test questions called benchmarks, and each model is scored based on the average of all its answers. But this paradigm has growing problems: it's expensive, time-consuming, and often includes poor-quality or redundant questions. These issues can distort the results and make it harder to trust what the scores really mean. In this paper, we propose a new approach inspired by how human abilities are measured: adaptive testing from the field of psychometrics. Instead of treating every question equally, we estimate how useful or difficult each one is and dynamically adjust the test for each AI system. This leads to a fairer, faster, and more accurate way to measure an AI’s true capabilities. We argue that this shift in evaluation paradigm will not only save time and resources, but also give us deeper insights into what modern AI systems can and can't do.

Primary Area: Research Priorities, Methodology, and Evaluation

Keywords: AI Evaluation, Adaptive Testing, Benchmark, Psychometrics

Submission Number: 43