Who Evaluates the Evaluators? Governance Challenges in AI Safety and Alignment Evaluations: A Review and Future Agenda

AAAI 2026 Workshop AIGOV Submission33 Authors

Keywords: AI Governance, AI Safety, Evaluation, Benchmarking, Meta-Governance, Systematic Review, Bibliometrics

TL;DR: This paper uses bibliometric analysis to reveal that AI safety benchmarks lack governance and proposes a governance of evaluation framework based on participation, transparency through Evaluation Cards, and institutional independence.

Abstract: AI safety and alignment evaluations (benchmarks) have become de facto governance tools in the rapidly evolving landscape of artificial intelligence. However, these critical instruments themselves lack systematic governance oversight, raising the central question: "Who evaluates the evaluators?" Through a systematic mapping of 298 papers and in-depth thematic analysis of 50 core studies, this paper identifies fundamental governance challenges in current AI evaluation practices. The bibliometric analysis reveals explosive growth in this field since 2023, with persistent gaps in transparency, accountability, and institutional independence. This position paper synthesizes current knowledge on evaluation governance and proposes a future agenda grounded in three core principles: participatory design with adversarial scrutiny, radical transparency through "Evaluation Cards," and institutional pluralism to ensure independent oversight. The paper argues for a paradigm shift from "evaluating AI" to "governing evaluations," offering actionable recommendations for researchers, policymakers, and practitioners to build a trustworthy evaluation ecosystem.

Submission Number: 33