Risk-Sensitive Theory of Mind: Coordinating with Agents of Unknown Bias using Cumulative Prospect Theory

Mason O. Smith, Wenlong Zhang

TL;DR: A novel framework called Risk-Sensitive Theory of Mind to coordinate with agents of unknown bias.

Abstract: Humans are often modeled as rational actors by interactive agents when they are in fact frequently observed to make biased decisions. This erroneous assumption may cause an agent’s model of the human to fail, especially when interaction occurs in bias-inducing settings that prompt risky decisions. To address this, this paper formulates a risk-sensitive multi-agent coordination problem and presents the novel Risk-Sensitive Theory of Mind (RS-ToM) framework that allows an autonomous agent to reason about and adapt to a partner of unknown risk-sensitivity. In simulated studies, we show that an agent with an RS-ToM is able to better coordinate with such a partner when compared to an agent that assumes their partner is rational. Thus, we observe significant improvements to team performance, coordination fluency, compliance with partner risk-preferences, and predictability. The presented results suggest that an RS-ToM will be able to model and plan with partners that exhibit these risk-sensitive biases in the real world.

Lay Summary: Machines often fail to consider that humans are not perfectly rational agents. This can lead to a machine that does not truly understand what their human partner wants or intends to do. Therefore, this paper proposes a machine learning framework called a Risky-Sensitive Theory of Mind (RS-ToM) that can model humans who are subject to risk-sensitive biases. In simulated experiments, we show that our approach can adapt to agents of varying and unknown risk-sensitivity in a way that the standard assumption of "humans are rational" fails to do. Consequently, the presented results suggest that an RS-ToM will be able to model and plan with partners that exhibit these risk-sensitive biases in the real world.

Link To Code: https://github.com/ASU-RISE-Lab/risky_overcooked/

Primary Area: Reinforcement Learning->Multi-agent

Keywords: Human-autonomy teaming, reinforcement learning, risk, cumulative prospect theory

Submission Number: 5340