Unveiling the latent dynamics in social cognition with multi-agent inverse reinforcement learning

Yusi Chen, Angela Radulescu, Herbert Zheng Wu

Keywords: Inverse Reinforcement Learning; Social Behavioral Latents; Theory of Mind Inference

TL;DR: We adopted multi-agent inverse reinforcement learning with value decomposition and recursive reasoning to infer goal maps and mutual prediction models in social cognition.

Abstract: Understanding the intentions and beliefs of others, a phenomenon known as "theory of mind", is a crucial element in social behavior. These beliefs and perceptions are inherently subjective and latent, making them often unobservable for investigation. Social interactions further complicate the matter, as multiple agents can engage in recursive reasoning about each other's strategies with increasing levels of cognitive hierarchy. While previous research has shown promise in understanding a single agent's belief of values through inverse reinforcement learning, extending this to model multiple agents remains an open challenge due to the computational complexity. In this work, we adopted a probabilistic recursive modeling of cognitive levels and joint value decomposition to achieve efficient multi-agent inverse reinforcement learning (MAIRL). We provided a numerical method to evaluate value decomposition errors in multi-agent tasks with discrete state and action spaces. To validate our method, we conducted simulations of a two-agent cooperative foraging task in a grid environment. Our algorithm revealed the ground truth goal-directed value function and effectively distinguished between level-0 and level-1 agents. When applied to human behavior in a cooperative hallway task, our method identified meaningful goal maps that evolved with task proficiency and an interaction map that is related to key states in the task without accessing to the task rules. Similarly, in a non-cooperative task performed by monkeys, we identified mutual predictions that correlated with the animals' social hierarchy, highlighting the behavioral relevance of the latent beliefs we uncovered. Together, our findings demonstrate that MAIRL offers a new framework for uncovering human or animal beliefs in social behavior, thereby illuminating previously opaque aspects of social cognition.

Primary Area: applications to neuroscience & cognitive science

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 9123