Generalization in Cooperative Multi-Agent Systems

TMLR Paper1331 Authors

Abstract:

Collective intelligence is a fundamental trait shared by many species that has allowed them
to thrive in diverse environmental conditions. From simple organisations in an ant colony
to complex systems in human groups, collective intelligence is vital for solving many survival
tasks. Such natural systems are flexible to changes in their structure: they generalize well
when the abilities or number of agents change, which we call Combinatorial Generalization
(CG). CG is a highly desirable trait for autonomous systems as it can increase their utility
and deployability across a wide range of applications. While recent works addressing
specific aspects of CG have shown impressive results on complex domains, they provide no
performance guarantees when generalizing to novel situations. In this work, we shed light on
the theoretical underpinnings of CG for cooperative multi-agent systems (MAS). Specifically,
we study generalization bounds under a linear dependence of the underlying dynamics on
the agent capabilities, which can be seen as a generalization of Successor Features to MAS.
We then extend the results first for Lipschitz and then arbitrary dependence of rewards on
team capabilities. Finally, empirical analysis on various domains using the framework of
multi-agent reinforcement learning highlights important desiderata for multi-agent algorithms
towards ensuring CG.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

1. Added more details and discussion for the theoretical results when there is non linear dependence of rewards and transition dynamics on the agent capabilities.
2. Added more discussion on the Experimental evaluation and Results (Sec 4,5)
3. Fixed minor typos

Assigned Action Editor: ~ERIC_EATON1

Submission Number: 1331