PettingZoo: Gym for Multi-Agent Reinforcement Learning

J K Terry, Benjamin J Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis Santos, Clemens Dieffendahl, Caroline Horsch, Rodrigo De Lazcano Perez-Vicente, Niall L Williams, Yashas Lokesh, Praveen Ravi

Keywords: Reinforcement Learning, Multi-agent Reinforcement Learning, Gym

TL;DR: We introduce PettingZoo, a package akin to Gym for multi-agent reinforcement learning, along with a novel model for computational multi-agent games

Abstract: This paper introduces the PettingZoo library and the accompanying Agent Environment Cycle (``"AEC") games model. PettingZoo is a library of diverse sets of multi-agent environments with a universal, elegant Python API. PettingZoo was developed with the goal of accelerating research in Multi-Agent Reinforcement Learning (``"MARL"), by making work more interchangeable, accessible and reproducible akin to what OpenAI's Gym library did for single-agent reinforcement learning. PettingZoo's API, while inheriting many features of Gym, is unique amongst MARL APIs in that it's based around the novel AEC games model. We argue, in part through case studies on major problems in popular MARL environments, that the popular game models are poor conceptual models of the games commonly used with MARL, that they promote severe bugs that are hard to detect, and that the AEC games model addresses these problems.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

Code: https://github.com/Farama-Foundation/PettingZoo

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2009.14471/code)