The Reflective Explorer: Online Meta-Exploration from Offline Data in Realistic Robotic Tasks

Rafael Rafailov, Varun Kumar Vijay, Tianhe Yu, Avi Singh, Mariano Phielipp, Chelsea Finn

Keywords: offline reinforcement learning, meta reinforcement learning, variational modelling, meta exploration, robot learning

TL;DR: We train a model to do meta-reinforcement learning using offline datasets for training and online exploration in a realistic robot setting

Abstract: Reinforcement learning is difficult to apply to real world problems due to high sample complexity, the need to adapt to frequent distribution shifts and the complexities of learning from high-dimensional inputs, such as images. Over the last several years, meta-learning has emerged as a promising approach to tackle these problems by explicitly training an agent to quickly adapt to new tasks. However, such methods still require huge amounts of data during training and are difficult to optimize in high-dimensional domains. One potential solution is to consider offline or batch meta-reinforcement learning (RL) - learning from existing datasets without additional environment interactions during training. In this work we develop the first offline model-based meta-RL algorithm that operates from images in tasks with sparse rewards. Our approach has three main components: a novel strategy to construct meta-exploration trajectories from offline data, which allows agents to learn meaningful meta-test time task inference strategy; representation learning via variational filtering and latent conservative model-free policy optimization. We show that our method completely solves a realistic meta-learning task involving robot manipulation, while naive combinations of previous approaches fail.