Learning Goal-Conditioned Policies Offline with Self-Supervised Reward ShapingDownload PDF

Published: 10 Sept 2022, Last Modified: 17 Sept 2023CoRL 2022 PosterReaders: Everyone
Student First Author: yes
Keywords: Offline Reinforcement Learning, Self-Supervised Learning, Goal-Conditioned RL
TL;DR: We propose a self-supervised reward shaping method for training goal-conditioned policies on pre-collected dataset without performing a single action in the environment.
Abstract: Developing agents that can execute multiple skills by learning from pre-collected datasets is an important problem in robotics, where online interaction with the environment is extremely time-consuming. Moreover, manually designing reward functions for every single desired skill is prohibitive. Prior works targeted these challenges by learning goal-conditioned policies from offline datasets without manually specified rewards, through hindsight relabeling. These methods suffer from the issue of sparsity of rewards, and fail at long-horizon tasks. In this work, we propose a novel self-supervised learning phase on the pre-collected dataset to understand the structure and the dynamics of the model, and shape a dense reward function for learning policies offline. We evaluate our method on three continuous control tasks, and show that our model significantly outperforms existing approaches, especially on tasks that involve long-term planning.
Supplementary Material: zip
Community Implementations: [![CatalyzeX](/images/catalyzex_icon.svg) 1 code implementation](https://www.catalyzex.com/paper/arxiv:2301.02099/code)
Website: https://linamezghani.github.io/go-fresh
13 Replies