Scaling Covariance Matrix Adaptation MAP-Annealing to High-Dimensional Controllers

Bryon Tjanaka, Matthew Christopher Fontaine, Aniruddha Kalkar, Stefanos Nikolaidis

Keywords: quality diversity, reinforcement learning

TL;DR: To train diverse robot controllers, we scale an existing high-performing quality diversity method which is based on evolution strategies.

Abstract: Pre-training a diverse set of robot controllers in simulation has enabled robots to adapt online to damage in robot locomotion tasks. However, finding diverse, high-performing controllers requires specialized hardware and extensive tuning of a large number of hyperparameters. On the other hand, the Covariance Matrix Adaptation MAP-Annealing algorithm, an evolution strategies (ES)-based quality diversity algorithm, does not have these limitations and has been shown to achieve state-of-the-art performance in standard benchmark domains. However, CMA-MAE cannot scale to modern neural network controllers due to its quadratic complexity. We leverage efficient approximation methods in ES to propose three new CMA-MAE variants that scale to very high dimensions. Our experiments show that the variants outperform ES-based baselines in benchmark robotic locomotion tasks, while being comparable with state-of-the-art deep reinforcement learning-based quality diversity algorithms. Source code and videos are available at https://scalingcmamae.github.io