Efficient Sparse-Reward Goal-Conditioned Reinforcement Learning with a High Replay Ratio and Regularization

TMLR Paper2017 Authors

Abstract:

Reinforcement learning (RL) methods with a high replay ratio (RR) and regularization have gained interest due to their superior sample efficiency.
However, these methods have mainly been developed for dense-reward tasks.
In this paper, we aim to extend these RL methods to sparse-reward goal-conditioned tasks.
We use Randomized Ensemble Double Q-learning (REDQ) (Chen et al., 2021), an RL method with a high RR and regularization.
To apply REDQ to sparse-reward goal-conditioned tasks, we make the following modifications to it:
(i) using hindsight experience replay and (ii) bounding target Q-values.
We evaluate REDQ with these modifications on 12 sparse-reward goal-conditioned tasks of Robotics (Plappert et al., 2018), and show that it achieves about $2 \times$ better sample efficiency than previous state-of-the-art (SoTA) RL methods.
Furthermore, we reconsider the necessity of specific components of REDQ and simplify it by removing unnecessary ones.
The simplified REDQ with our modifications achieves $\sim 8 \times$ better sample efficiency than the SoTA methods in 4 Fetch tasks of Robotics.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

While a previous version of this submission was rejected by TMLR, we cannot provide the link because the original TMLR submission is not anonymized.

Our previous submission was rejected because it contains the Google Drive link to the file containing the authors' information.
To address this, we removed the Google Drive link from the submission.

Original review comment from the action editor:
The submitted paper is unfortunately not anonymous. The identity of the author is indirectly disclosed through the google drive link provided in the caption of figure 1 (the video is hosted on the personal google drive of the author, whose identity is disclosed by checking the details of the shared file).

Assigned Action Editor: ~Marc_Lanctot1

Submission Number: 2017