Unexploited Information Value in Human-AI Collaboration

Published: 10 Oct 2024, Last Modified: 01 Nov 2024NeurIPS 2024 Workshop on Behavioral MLEveryoneRevisionsBibTeXCC BY 4.0
Keywords: decision making, human-AI collaboration, rational agent
Abstract: Humans and AIs are often paired on decision tasks with the expectation of achieving complementary performance -- where the combination of human and AI outper- forms either one alone. However, how to improve performance of a human-AI team is often not clear without knowing more about what particular information and strategies each agent employs. In this paper, we propose a model based in statistically decision theory to analyze human-AI collaboration from the perspec- tive of what information could be used to improve a human or AI decision. We demonstrate our model on a deepfake detection task to investigate seven video-level features by their unexploited value of information. We compare the human alone, AI alone and human-AI team and offer insights on how the AI assistance impacts people’s usage of the information and what information that the AI exploits well might be useful for improving human decisions.
Submission Number: 74
Loading

OpenReview is a long-term project to advance science through improved peer review with legal nonprofit status. We gratefully acknowledge the support of the OpenReview Sponsors. © 2025 OpenReview