SoK: AI Auditing: The Broken Bus on the Road to AI Accountability

Abeba Birhane, Ryan Steed, Victor Ojewale, Briana Vecchione, Inioluwa Deborah Raji

Keywords: evaluation, audits, accountability

Abstract:

One of the most concrete measures to take towards
meaningful AI accountability is to consequentially assess and
report the systems’ performance and impact. However, the
practical nature of the “AI audit” ecosystem is muddled and
imprecise, difficult to work through various concepts, and map
out the stakeholders involved in the practice. First, we taxonomize
current AI audit practices as completed by regulators, law firms,
civil society, journalism, academia, and consulting agencies. Next,
we assess the impact of audits done by stakeholders within each
domain. We find that only a subset of AI audit studies translate
to the desired accountability outcomes. We thus assess and isolate
practices necessary for effective AI audit results, articulating
the observed connections between AI audit design, methodology
and institutional context on its effectiveness as a meaningful
mechanism for accountability.

Submission Number: 40