Decoding Rewards in Competitive Games: Inverse Game Theory with Entropy Regularization

Junyi Liao, Zihan Zhu, Ethan X Fang, Zhuoran Yang, Vahid Tarokh

TL;DR: his paper introduces a unified framework for reward function recovery in two-player zero-sum matrix games and Markov games with entropy regularization.

Abstract: Estimating the unknown reward functions driving agents' behavior is a central challenge in inverse games and reinforcement learning. This paper introduces a unified framework for reward function recovery in two-player zero-sum matrix games and Markov games with entropy regularization. Given observed player strategies and actions, we aim to reconstruct the underlying reward functions. This task is challenging due to the inherent ambiguity of inverse problems, the non-uniqueness of feasible rewards, and limited observational data coverage. To address these challenges, we establish reward function identifiability using the quantal response equilibrium (QRE) under linear assumptions. Building on this theoretical foundation, we propose an algorithm to learn reward from observed actions, designed to capture all plausible reward parameters by constructing confidence sets. Our algorithm works in both static and dynamic settings and is adaptable to incorporate other methods, such as Maximum Likelihood Estimation (MLE). We provide strong theoretical guarantees for the reliability and sample-efficiency of our algorithm. Empirical results demonstrate the framework’s effectiveness in accurately recovering reward functions across various scenarios, offering new insights into decision-making in competitive environments.

Lay Summary:

Understanding how and why people or machines make decisions is a central goal in fields like economics, robotics, and artificial intelligence. This paper studies how to uncover the hidden goals (or "rewards") of decision-makers when we only observe their behavior in competitive settings—like games or strategic interactions—where two sides are working against each other. We focus on situations where both players make decisions with some randomness, which is more realistic than assuming perfect reasoning.

Our main contribution is a method to learn these hidden goals from data, even when we can't pinpoint one unique answer. Instead of guessing a single explanation, we build a set of all plausible explanations that fit the observed behavior. We also show, with both theory and experiments, how more data leads to better understanding of the decision-makers’ goals. Our work provides new tools for safely and reliably interpreting strategic behavior in complex systems, such as autonomous driving or financial markets.

Primary Area: Reinforcement Learning->Inverse

Keywords: Inverse game; Quantal Response Equilibrium; Reward recovery; confidence sets

Submission Number: 12155