Finite-Time Convergence Rates in Stochastic Stackelberg Games with Smooth Algorithmic Agents

Eric Frankel, Kshitij Kulkarni, Dmitriy Drusvyatskiy, Sewoong Oh, Lillian J. Ratliff

TL;DR: Finite-time convergence rates for stochastic Stackelberg games under distribution shifts resultant from reactive agents employing smooth algorithms.

Abstract:

Decision-makers often adaptively influence downstream competitive agents' behavior to minimize their cost, yet in doing so face critical challenges: $(i)$ decision-makers might not *a priori* know the agents' objectives; $(ii)$ agents might *learn* their responses, introducing stochasticity and non-stationarity into the decision-making process; and $(iii)$ there may be additional non-strategic environmental stochasticity.
Characterizing convergence of this complex system is contingent on how the decision-maker controls for the tradeoff between the induced drift and additional noise from the learning agent behavior and environmental stochasticity. To understand how the learning agents' behavior is influenced by the decision-maker’s actions, we first consider a decision-maker that deploys an arbitrary sequence of actions which induces a sequence of games and corresponding equilibria. We characterize how the drift and noise in the agents' stochastic algorithms decouples from their optimization error. Leveraging this decoupling and accompanying finite-time efficiency estimates, we design decision-maker algorithms that control the induced drift relative to the agent noise. This enables efficient finite-time tracking of game theoretic equilibrium concepts that adhere to the incentives of the players' collective learning processes.

Lay Summary:

Many modern problems to which machine learning is applied are such that a decision-maker takes consequential actions that influence downstream user behavior or subsequent actions. Decision-makers in such settings are adaptively shaping downstream competitive agents' behavior in order to minimize their cost, yet in doing so face critical challenges: $(i)$ decision-makers might not *a priori* know the agents' objectives; $(ii)$ agents might *learn* their response, introducing stochasticity into the decision-making process; and $(iii)$ there may be additional non-strategic environmental stochasticity. Characterizing convergence of this complex system is contingent on how the decision-maker accounts for the tradeoff in the "drift" they induce in the agents' learning processes and the noise from the learning agents' behavior and the environmental stochasticity.

To understand how the learning agents' behavior is influenced by the decision-maker’s actions, we consider a decision-maker that deploys an (potentially arbitrary) sequence of actions each of which induces a game amongst the agents and corresponding equilibria. We characterize how the drift and noise in the agents' stochastic algorithms decouples from their optimization error. Leveraging this decoupling and accompanying efficiency estimates, we design decision-maker algorithms that control the induced drift relative to the agent noise. Further, we establish a hierarchy of reasonable interaction models that allow progressively more gradient information on the part of the decision-maker, and synthesize appropriate algorithms. This enables efficient finite-time tracking of game theoretic equilibrium concepts that are meaningful in the sense that they respect the incentive structure of the players' learning processes.

Link To Code: https://github.com/SewoongLab/stoch-stackelberg

Primary Area: Theory->Game Theory

Keywords: Stackelberg game, stochastic optimization, learning and games, stochastic tracking

Submission Number: 7670