Position: Measuring Bias in AI Agents Is (Still) a Construct Validity Problem

Soubhik Barari, Diwas Bhattarai, Joshua Y. Lerner

Keywords: AI Bias, Construct Validity, Computational Social Science, AI Evaluation

TL;DR: Measuring bias in AI agents is complicated by a new form of bias not present in previous AI systems, but still requires careful social science measurement principles.

Abstract: The rapid deployment of AI systems has intensified concerns about bias. Yet "bias" remains loosely defined in the AI evaluation literature, often collapsing distinct phenomena that require different measurement strategies. Drawing on social science research, we propose a framework that (1) distinguishes three dimensions of bias, (2) separates how bias appears from the processes and evaluation choices that produce biased behavior or biased inferences about it, and (3) explains how agentic systems complicate bias through delegation. We argue that rigorous bias evaluation requires explicit construct definition, multiple operationalizations, validity evidence, and uncertainty-aware robustness analysis, especially as AI systems evolve from static language models to autonomous agents.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Type: Provocation

Archival Status: Non-archival

Submission Number: 81