Variational Inequality Methods for Multi-Agent Reinforcement Learning: Performance and Stability Gains

Baraah A. M. Sidahmed, Tatjana Chavdarova

Keywords: multi-agent reinforcement learning, game optimization, Variational Inequality

Abstract:

Multi-agent reinforcement learning (MARL) poses distinct challenges as agents learn strategies through experiences. Gradient-based methods often fail to converge in MARL, and performances are highly sensitive to initial random seeds, contributing to what has been termed the MARL reproducibility crisis.
Concurrently, significant advances have been made in solving Variational Inequalities (VIs)---which include equilibrium-finding problems---particularly in addressing the non-converging rotational dynamics that impede convergence of traditional gradient-based optimization methods.
This paper explores the potential of leveraging VI-based techniques to improve MARL training. Specifically, we study the integration of VI methods---namely, Nested-Lookahead VI (nLA-VI) and Extragradient (EG)---into the multi-agent deep deterministic policy gradient (MADDPG) algorithm.
We present a VI reformulation of the actor-critic algorithm for both single- and multi-agent settings. We introduce three algorithms that use nLA-VI, EG, and a combination of both, named LA-MADDPG, EG-MADDPG, and LA-EG-MADDPG, respectively.
Our empirical results show that these VI-based approaches yield significant performance improvements in benchmark environments, such as the zero-sum games: rock-paper-scissors and matching pennies, where equilibrium strategies can be quantitatively assessed, and the Multi-Agent Particle Environment: Predator-prey benchmark, where VI-based methods also yield balanced participation of agents from the same team, further highlighting the substantial impact of advanced optimization techniques on MARL performance.

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 11997