Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels

Denis Yarats, Ilya Kostrikov, Rob Fergus

Abstract: We propose a simple data augmentation technique that can be applied to standard model-free reinforcement learning algorithms, enabling robust learning directly from pixels without the need for auxiliary losses or pre-training. The approach leverages input perturbations commonly used in computer vision tasks to transform input examples, as well as regularizing the value function and policy. Existing model-free approaches, such as Soft Actor-Critic (SAC), are not able to train deep networks effectively from image pixels. However, the addition of our augmentation method dramatically improves SAC’s performance, enabling it to reach state-of-the-art performance on the DeepMind control suite, surpassing model-based (Hafner et al., 2019; Lee et al., 2019; Hafner et al., 2018) methods and recently proposed contrastive learning (Srinivas et al., 2020). Our approach, which we dub DrQ: Data-regularized Q, can be combined with any model-free reinforcement learning algorithm. We further demonstrate this by applying it to DQN and significantly improve its data-efficiency on the Atari 100k benchmark.

One-sentence Summary: The first successful demonstration that image augmentation can be applied to image-based Deep RL to achieve SOTA performance.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Code: [ denisyarats/drq](https://github.com/denisyarats/drq) + [ 3 community implementations](https://paperswithcode.com/paper/?openreview=GY6-6sTvGaf)

Data: [Atari 100k](https://paperswithcode.com/dataset/atari-100k), [DeepMind Control Suite](https://paperswithcode.com/dataset/deepmind-control-suite)

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2004.13649/code)