The impact of uncertainty on regularized learning in games

Pierre-Louis Cauvin, Davide Legacci, Panayotis Mertikopoulos

TL;DR: We examine how randomness and uncertainty influence learning in games

Abstract:

In this paper, we investigate how randomness and uncertainty influence learning in games. Specifically, we examine a perturbed variant of the dynamics of “follow-the-regularized-leader” (FTRL), where the players’ payoff observations and strategy updates are continually impacted by random shocks. Our findings reveal that, in a fairly precise sense, “uncertainty favors extremes”: in any game, regardless of the noise level, every player’s trajectory of play reaches an arbitrarily small neighborhood of a pure strategy in finite time (which we estimate). Moreover, even if the player does not ultimately settle at this strategy, they return arbitrarily close to some
(possibly different) pure strategy infinitely often. This prompts the question of which sets of pure strategies emerge as robust predictions of learning under uncertainty. We show that (a) the only possible limits of the FTRL dynamics under uncertainty are pure Nash equilibria; and (b) a span of pure strategies is stable and attracting if and only if it is closed under better replies. Finally, we turn to games where the deterministic dynamics are recurrent—such as zero-sum games with interior equilibria—and show that randomness disrupts this behavior, causing the stochastic dynamics to drift toward the boundary on average.

Lay Summary:

We examine what happens when decision-makers are involved in a repeated decision process that repeats over time—players interact, compete, and make decisions, each aiming to maximize their individual rewards. The specific question we're asking is what happens in the presence of noise, randomness and uncertainty, that is, when players have a very hazy vision of their environment, and/or the behavior and objectives of other players in the game.

In this case, we find that uncertainty favors extremes: after some time, players act in rigid, uncompromising ways. Yet, even in the face of uncertainty, not all rationality is lost to chance, and some patterns can still be anticipated: one can predict which extreme behaviors are more likely to emerge from the range of possible ones. In particular, players still gravitate toward behaviors that, while extreme, reflect the best possible compromise given the circumstances. However, such a compromise isn't always possible. In some scenarios, no stable agreement exists, and the players alternate between extremes, without ever getting close to a stable, equilibrium state.

Primary Area: Theory->Game Theory

Keywords: Follow the regularized leader, uncertainty, Nash equilibrium

Submission Number: 10614