rQdia: Regularizing Q-Value Distributions With Image Augmentation

Samuel Lerman, Jing Bi, Chenliang Xu

Keywords: deep reinforcement learning, regularization, q-value distributions, invariance, image augmentation, continuous control, Atari

Abstract: rQdia (pronounced “Arcadia”) regularizes Q-value distributions with augmented images in pixel-based deep reinforcement learning. With a simple auxiliary loss, that equalizes these distributions via MSE, rQdia boosts DrQ and SAC on 9/12 and 10/12 tasks respectively in the MuJoCo Continuous Control Suite from pixels, and Data-Efficient Rainbow on 18/26 Atari Arcade environments. Gains are measured in both sample efficiency and longer-term training. Moreover, the addition of rQdia finally propels model-free continuous control from pixels over the state encoding baseline. Additional results, namely more random seeds, pending.

One-sentence Summary: rQdia (pronounced “Arcadia”) regularizes Q-value distributions with augmented images in pixel-based deep reinforcement learning.