Detecting Influence Structures in Multi-Agent Reinforcement Learning

Fabian Raoul Pieroth, Katherine Fitch, Lenz Belzner

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: multi-agent reinforcement learning, multi-agent interdependencies, stochastic approximation, decentralized algorithms

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: Introducing novel, unified metrics and decentralized algorithms in MARL to precisely quantify agent influence, with empirical validation and convergence guarantees.

Abstract:

We consider the problem of quantifying the amount of influence one agent can exert on another in the setting of multi-agent reinforcement learning (MARL).
As a step towards a unified approach to express agents' interdependencies, we introduce the total and state influence measurement functions.
Both of these are valid for all common MARL systems, such as the discounted reward setting.
Additionally, we propose novel quantities, called the total impact measurement (TIM) and state impact measurement (SIM), that characterize one agent's influence on another by the maximum impact it can have on the other agents' expected returns and represent instances of impact measurement functions in the average reward setting.
Furthermore, we provide approximation algorithms for TIM and SIM with simultaneously learning approximations of agents' expected returns, error bounds, stability analyses under changes of the policies, and convergence guarantees.
The approximation algorithm relies only on observing other agents' actions and is, other than that, fully decentralized.
Through empirical studies, we validate our approach's effectiveness in identifying intricate influence structures in complex interactions.
Our work appears to be the first study of determining influence structures in the multi-agent average reward setting with convergence guarantees.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6381