Decoupled Stochastic Gradient Descent for N-Player Games

Ali Zindari, Parham Yazdkhasti, Tatjana Chavdarova, Sebastian U Stich

Keywords: optimization, multi agent RL

Abstract:

We study games with $N$ players where each player aims to minimize their own loss. These games are gaining popularity due to their wide range of applications in machine learning. For instance, minimax optimization problems are a special case of $N$-player games. Stochastic Gradient Descent (SGD) is among the main methods for solving such games. However, in many distributed game optimization applications, this approach can result in high communication overhead, as each player needs access to the other players' strategies at every time step to compute a gradient.

In this paper, we introduce a new optimization paradigm called \emph{Decoupled SGD}. This framework allows individual players to carry out SGD updates independently, with occasional strategy exchanges at predetermined intervals. We analyze the convergence properties of this approach in various scenarios.

Primarily, we consider the popular minimax bi-linear game and establish the convergence rate of our method in this setting. We also derive explicit formulas for the optimal length of synchronization intervals and step size.
We then provide a general algorithm for $N$-player games for cases where strategy synchronization is costly. We derive its convergence rate when the resulting operator is strongly monotone.
Finally, for minimax optimization problems, we investigate the combination of our Decoupled SGD with classical distributed paradigms, where players have multiple processors/clients and synchronize their strategies sporadically.

Submission Number: 110