Multi-Agent Reinforcement Learning of Karma Bidding Strategies

Kevin Riehl, Anastasia Psarou, Robert Müller, Fan Wu, Patrick Langer, Robert Jakob, Gabor Hollbeck, Anastasios Kouvelas, Rafal Kucharski, Michail A. Makridis

Keywords: Resource allocation mechanism, Karma, multi-agent reinforcement learning

TL;DR: We study suitable regimes to prove convergence towards stochastic Nash equilibria when following a multi-agent reinforcement learning approach in Karma games.

Abstract: Capacity-constrained shared infrastructure systems require demand management mechanisms that balance efficiency and fairness. Karma mechanisms address this challenge using an artificial, non-tradable currency that enables decentralized allocation through repeated bidding, but computing equilibrium strategies in such settings is difficult due to unknown population dynamics, stochastic demand, and computation time in practice. Here, we study suitable regimes to prove convergence towards stationary Nash equilibria when following a multi-agent reinforcement learning approach. We develop a conditional proof for convergence to entropy-regularized equilibria, and discuss convergence to unregularized ones in a limit. Computational case studies in a controlled synthetic benchmark demonstrate empirically that learned policies closely approximate equilibrium behavior, and further assess impact of learning algorithm and policy initialization on convergence speed. This work highlights the practical potential of the approach for real-world decision support in repeated access-allocation settings.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Paper Type: Standard paper

Submission Number: 22