Learning Controllable Elements Oriented Representations for Reinforcement Learning

Qi Yi, Jiaming Guo, Rui Zhang, Shaohui Peng, Xing Hu, Xishan Zhang, Ke Tang, Zidong Du, Qi Guo, Yunji Chen

Keywords: reinforcement learning, representation learning, mutual information

Abstract: Deep Reinforcement Learning (deep RL) has been successfully applied to solve various decision-making problems in recent years. However, the observations in many real-world tasks are often high dimensional and include much task-irrelevant information, limiting the applications of RL algorithms. To tackle this problem, we propose LCER, a representation learning method that aims to provide RL algorithms with compact and sufficient descriptions of the original observations. Specifically, LCER trains representations to retain the controllable elements of the environment, which can reflect the action-related environment dynamics and thus are likely to be task-relevant. We demonstrate the strength of LCER on the DMControl Suite, proving that it can achieve state-of-the-art performance. To the best of our knowledge, LCER is the first representation learning algorithm that enables the pixel-based SAC to outperform state-based SAC on the DMControl 100K benchmark, showing that the obtained representations can match the oracle descriptions ($i.e.$ the physical states) of the environment.

One-sentence Summary: We train representations for RL by encoding the controllable elements of the environment.