Adaptable Agent Populations via a Generative Model of Policies

Kenneth Derek, Phillip Isola

Keywords: reinforcement learning, quality diversity, multiagent, generative models, latent policy

Abstract: In the natural world, life has found innumerable ways to survive and often thrive. Between and even within species, each individual is in some manner unique, and this diversity lends adaptability and robustness to life. In this work, we aim to learn a space of diverse and high-reward policies in a given environment. To this end, we introduce a generative model of policies for reinforcement learning, which maps a low-dimensional latent space to an agent policy space. Our method enables learning an entire population of agent policies, without requiring the use of separate policy parameters. Just as real world populations can adapt and evolve via natural selection, our method is able to adapt to changes in our environment solely by selecting for policies in latent space. We test our generative model’s capabilities in a variety of environments, including an open-ended grid-world and a two-player soccer environment. Code, visualizations, and additional experiments can be found at https://kennyderek.github.io/adap/.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

TL;DR: We learn a multi-modal policy space in a reinforcement learning setting that creates diverse and adaptable agent populations.

Code: https://github.com/kennyderek/adap

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2107.07506/code)