Robust Reinforcement Learning using Adversarial Populations

Eugene Vinitsky, Yuqing du, Kanaad V Parvate, Kathy Jang, Pieter Abbeel, Alexandre Bayen

Keywords: Robust Control, Reinforcement Learning, Multiagent Systems

Abstract: Reinforcement Learning (RL) is an effective tool for controller design but can struggle with issues of robustness, failing catastrophically when the underlying system dynamics are perturbed. The Robust RL formulation tackles this by adding worst-case adversarial noise to the dynamics and constructing the noise distribution as the solution to a zero-sum minimax game. However, existing work on learning solutions to the Robust RL formulation has primarily focused on training a single RL agent against a single adversary. In this work, we demonstrate that using a single adversary does not consistently yield robustness to dynamics variations under standard parametrizations of the adversary; the resulting policy is highly exploitable by new adversaries. We propose a population-based augmentation to the Robust RL formulation in which we randomly initialize a population of adversaries and sample from the population uniformly during training. We empirically validate across a variety of benchmarks that the use of an adversarial population results in a less exploitable, more robust policy. Finally, we demonstrate that this approach provides comparable robustness and generalization as domain randomization on these benchmarks while avoiding a ubiquitous domain randomization failure mode.

One-sentence Summary: We demonstrate that the standard robust RL formulation does not consistently yield robustness but that this can be alleviated using a population of adversaries.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2008.01825/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=r3UNEA7hm