Artificial Theory of Mind in Human-in-the-Loop

Sammie Katt, Samuel Kaski

Keywords: theory of mind, human-in-the-loop, human-ai collaboration, user modeling

TL;DR: A framework for modeling user's artificial theory of mind in human-in-the-loop problems and how this should be adapted to.

Abstract: Thanks to the advances in artificial intelligence (AI), interactive human-AI applications are growing explosively. A common assumption in these systems is that the humans provide ground-truth (oracle) data during interactions. This is seen, for example, when fine-tuning large language models with human feedback or in personalized recommendation-systems. However, it is well-known that human users often do not act like oracles which implies they, instead, should be represented more realistically instead. In this work, we propose a preliminary framework for user models for human-in-the-loop (HITL) problems. In particular, we first define a general decision-making problem statement for HITL which explicitly includes user models, with a focus on how they may reason about the AI they are interacting with. We then derive user models for HITL from simple but powerful assumptions about the user, and show the implications empirically in a Bayesian optimization and recommendation system setting. Through this lens, we discuss how assuming humans are oracles can lead to bias under several concrete settings.

Submission Number: 14