Enhancing Visual Generalization in Reinforcement Learning with Cycling Augmentation

Shengjie Sun, Jiafei Lyu, Lu Li, Jiazhe Guo, Mengbei Yan, Runze Liu, Xiu Li

Abstract: Effectively generalizing learned policies to unseen environments remains challenging in Visual Reinforcement Learning (Visual RL). Data Augmentation (DA) is widely used in existing approaches for better generalization, but it is often found to reduce sample efficiency and even lead to divergence. In this paper, we investigate the impact of Cycling Augmentation (CycAug) on Visual RL algorithms. We find that augmenting data with CycAug markedly enhances the generalization capabilities of contemporary algorithms in Visual RL and promotes their sample efficiency. Hence, we propose a simple yet effective framework to enhance the generalization capabilities of off-the-shelf visual RL algorithms through seamless integration with CycAug. We perform extensive experiments on the Reinforcement Learning Benchmark for Visual Generalization (RL-ViGen). Empirical evidence suggests that our framework improves performance significantly better than the original DA operation of the algorithms. In original scenarios of Locomotion and Dexterous Manipulation, algorithms empowered by CycAug consistently demonstrate superior or equivalent sample efficiency across various tasks. Notably, in diverse generalization scenarios, CycAug boosts the generalization capabilities of the SVEA method and the state-of-the-art PIE-G algorithm by an average of +28.6% and +5.0%, respectively. Our code is publicly available at: https://github.com/ShengjieSun419/Visual-RL-Generalization-with-CycAug. https://github.com/ShengjieSun419/Visual-RL-Generalization-with-CycAug.