Nash Equilibria in Reward-Potential Markov Games: Algorithms, Complexity, and Applications
Keywords: potential games, markov games, ppad-hardness, complexity, nash equilibrium, fisher market
TL;DR: Markov games that are potential in every stage game are hard; nevertheless, if the transitions are structured we can solve them efficiently.
Abstract: Markov games that exhibit potential functions for rewards in each state, referred to as Reward-Potential Markov Games (RPMGs), do not inherently qualify as Markov Potential Games (MPGs), which require state-dependent potential functions for value functions. This discrepancy, widely acknowledged in recent literature on MPGs, remains highly unexplored. RPMGs, with their easier-to-verify and arguably more minimal reward-potential property, have not received adequate attention. We embark on the exploration of RPMGs, observing that computing a stationary Nash equilibrium (NE) is $\mathsf{PPAD}$-hard for infinite-horizon RPMGs, even under constraints on transition functions. In contrast to results on stationary equilibria in Markov games, we establish that computing a nonstationary Nash equilibrium in finite-horizon RPMGs is $\mathsf{PPAD}$-hard without any assumptions on transition functions. On a positive note, we present an algorithm capable of breaking the curse of multiagents by efficiently computing an $\epsilon$-approximate NE in RPMGs with additive transitions, with a runtime polynomial in $1/\epsilon$. Furthermore, we extend our analysis to include an adversarial player seeking to maximize the underlying potential function, introducing the concept of Adversarial Reward-Potential Markov Games.
Primary Area: learning theory
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 5636
Loading