The Value of Information in Human-AI Decision-making

Ziyang Guo, Yifan Wu, Jason Hartline, Jessica Hullman

Keywords: Explanation, Human-AI complementarity, Decision theory

TL;DR: We emphasize the value of complementary information in AI-assisted human decision making.

Abstract:

Multiple agents are increasingly combined to make decisions with the expectation of achieving *complementary performance*, where the decisions they make together outperform those made individually.
However, knowing how to improve the performance of collaborating agents requires knowing what information and strategies each agent employs.
With a focus on human-AI pairings, we contribute a decision-theoretic framework for characterizing the value of information.
By defining complementary information, our approach identifies opportunities for agents to better exploit available information in AI-assisted decision workflows.
We present a novel explanation technique (ILIV-SHAP) that adapts SHAP explanations to highlight human-complementing information.
We validate the effectiveness of our framework and ILIV-SHAP through a study of human-AI decision-making, and demonstrate the framework on examples from chest X-ray diagnosis and deepfake detection.
We find that presenting ILIV-SHAP with AI predictions leads to reliably greater reductions in error over non-AI assisted decisions more
than vanilla SHAP.

Primary Area: interpretability and explainable AI

Submission Number: 14178