Collective Intelligence in Human-AI Teams: A Bayesian Theory of Mind Approach

Christoph Riedl, Samuel Westby

Abstract:

We develop a network of Bayesian agents that collectively
model the mental states of teammates from the observed com-
munication. Using a generative computational approach to
cognition, we make two contributions. First, we show that
our agent could generate interventions that improve the col-
lective intelligence of a human-AI team beyond what humans
alone would achieve. Second, we develop a real-time mea-
sure of human’s theory of mind ability and test theories about
human cognition. We use data collected from an online ex-
periment in which 145 individuals in 29 human-only teams
of five communicate through a chat-based system to solve a
cognitive task. We find that humans (a) struggle to fully in-
tegrate information from teammates into their decisions, es-
pecially when communication load is high, and (b) have cog-
nitive biases which lead them to underweight certain useful,
but ambiguous, information. Our theory of mind ability mea-
sure predicts both individual- and team-level performance.
Observing teams’ first 25% of messages explains about 8%
of the variation in final team performance, a 170% improve-
ment compared to the current state of the art.