Transparency for Agentic AI: A Study of Benchmarks, Metrics, and Oversight

ACL ARR 2026 January Submission2484 Authors

Keywords: Transparency, Auditability, Agentic AI, Evaluation, Benchmarking, Trustworthiness

Abstract: Agentic AI is an emerging class of systems that perceive, plan, and act to achieve complex real-world goals (e.g., web navigation, gaming, and software engineering). However, the the transparency of these systems is fragile because behavior unfolds across multi-step trajectories and interacting sub-agents; a failure in a single component propagate and degrade the whole execution. This study reviews benchmarking, evaluation, and governance approaches for agentic AI through a transparency lens. Existing surveys largely focus on agent architectures and capabilities; in contrast, we organize the literature around transparency and show that outcome-only evaluation fails to support auditability. To address this gap, we outline trace-based signals and metrics that better capture agent behavior across the full execution trajectory. Overall, we aim to provide a practical resource for researchers and practitioners and to support the development of more transparent and auditable agentic AI systems.

Paper Type: Long

Research Area: Ethics, Bias, and Fairness

Research Area Keywords: Ethics, Bias, and Fairness

Contribution Types: Surveys

Languages Studied: N/A

Submission Number: 2484