Unexploited Information Value in Human-AI Collaboration

Ziyang Guo, Yifan Wu, Jason Hartline, Jessica Hullman

Keywords: decision making, human-AI collaboration, rational agent

Abstract:

Humans and AIs are often paired on decision tasks with the expectation of achieving
complementary performance -- where the combination of human and AI outper-
forms either one alone. However, how to improve performance of a human-AI
team is often not clear without knowing more about what particular information
and strategies each agent employs. In this paper, we propose a model based in
statistically decision theory to analyze human-AI collaboration from the perspec-
tive of what information could be used to improve a human or AI decision. We
demonstrate our model on a deepfake detection task to investigate seven video-level
features by their unexploited value of information. We compare the human alone,
AI alone and human-AI team and offer insights on how the AI assistance impacts
people’s usage of the information and what information that the AI exploits well
might be useful for improving human decisions.

Submission Number: 74