The Impact of Explanations on Fairness in Human-AI Decision Making: Protected vs Proxy Features

Navita Goyal, Connor Baumler, Tin Trung Nguyen, Hal Daumé

Abstract: AI systems have been known to amplify biases in real world data. Human-AI teams have the potential to control for these biases for fairer decision-making, and there is hope that explanations can help humans understand and combat model biases. Traditionally, explanations focus on the input features that are salient to the model’s predictions. If a model is biased against some protected group, explanations may include features that demonstrate this bias. However, the relationship between a proxy feature and the protected one may be less clear to a human. In this work, we consider whether explanations are sufficient to alleviate model biases due to proxy features in human-AI decision-making teams. We study the effect of the presence of protected and proxy features on participants' perception of model fairness and their ability to improve demographic parity over an AI-only model. Further, we examine how different interventions—model bias disclosure and proxy correlation disclosure—affect fairness perception and parity. We find that explanations alone are not sufficient in flagging fairness concerns when model biases are caused by a proxy feature. However, proxy correlation disclosure helps participants identify unfairness and better decide when to rely on model predictions.