Learning to Cooperate with Humans through Theory-Informed Trust Beliefs

ICLR 2026 Conference Submission18106 Authors

Keywords: Human-AI Cooperation, Cooperative AI, Trust Modeling, RL Agent

Abstract:

Real-world human--AI cooperation is challenging due to the wide range of interests and capabilities that each party brings.
To maximize joint performance, cooperative AI must adapt its policies to the competence and incentives of its specific human partner.
Prevailing approaches address this challenge by training on human data or simulated partners. In this paper, we pursue an orthogonal approach: grounded on theory from social science, we hypothesize that equipping agents with human-like trust beliefs enables them to adapt as efficiently as humans do. We formulate the agent's problem as TrustPOMDP, a variant of POMPDs, and develop a trust model that captures three key factors known to shape human trust beliefs: ability, benevolence, and integrity (ABI). A key advantage of the approach is that it only requires minimal modifications to a POMDP agent. TrustPOMDPs can be trained with real or simulated partners, provided sufficient diversity in the three dimensions. Results from both simulated and human-subject experiments (N=102) show that TrustPOMDP-based agents adapt more rapidly and effectively, even to malevolent behavior, while baselines methods tend to over- or undertrust, reducing team performance. These findings highlight the promise of incorporating social science-informed trust models into RL agents to advance collaboration with humans.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 18106