Speeding up Meta-exploration via Latent Representation

BingCheng He, Han Wang, Qingshan Li

Abstract: Exploration poses a formidable challenge in reinforcement learning, especially within high-dimensional systems. Many prevailing exploration techniques take a task-agnostic approach, operating under the assumption that exploration occurs without any pre-existing knowledge regarding on task solutions. However, such assumption hinders adaptation to novel tasks, as the learning agent lacks prior insight into which actions might lead to failure states, with such failures being a future inevitability. In this work, we propose to take a task-relationship learning viewpoint on exploration. We aim to derive latent representations from historical trajectories acquired from previous tasks, which then serve to guide exploration in new tasks. Prior trajectories from related tasks preserve task relationships and immediate rewards can be used to generate informative state representations. Such latent representations can then be integrated with entropy-based exploration to effectively compress the state space. Furthermore, this paper demonstrates the enhanced performance of our relationship aware representation learning over previous meta-exploration methods across 50 robotic learning environments.