Teaching an Old Dynamics New Tricks: Regularization-free Last-iterate Convergence in Zero-sum Games via BNN Dynamics

Tuo Zhang, Leonardo Stella

Keywords: Multi-agent learning, Evolutionary dynamics, Game theory, Last-iterate convergence

Abstract: Zero-sum games are a fundamental setting for adversarial training and decision-making in multi-agent learning (MAL). Existing methods often ensure convergence to (approximate) Nash equilibria by introducing a form of regularization. Yet, regularization leads to the requirement of tuning additional hyperparameters, which is challenging even when the payoff structure is known, and becomes considerably harder when the structure is unknown or subject to change. Motivated by this problem, we repurpose a classical model in evolutionary game theory, i.e., the Brown–von Neumann–Nash (BNN) dynamics, into the MAL context where it has never been applied before. By doing so, we leverage the intrinsic properties of BNN dynamics to converge naturally in zero-sum games without the need for any form of regularization, and provide last-iterate convergence guarantees in noisy normal-form games. Importantly, to make this approach more broadly applicable, we develop a novel framework with theoretical guarantees that integrates BNN dynamics in extensive-form games through counterfactual weighting. Furthermore, we implement an algorithm that instantiates our framework with neural function approximation, enabling scalable learning in normal-form and extensive-form games. Empirical results show that our method quickly adapts to nonstationarities, outperforming the state-of-the-art regularization-based approach.

Area: Learning and Adaptation (LEARN)

Generative A I: I acknowledge that I have read and will follow this policy.

Submission Number: 1279