Hindsight Curriculum Generation Based Multi-Goal Experience Replay

Xiaoyun Feng

Abstract: In multi-goal tasks, an agent learns to achieve diverse goals from past experiences. Hindsight Experience Replay (HER)—which replays experiences with pseudo goals—has shown the potential to learn from failed experiences. However, not all the pseudo goals are well-explored to provide reliable value estimates. In view of value estimation, the agent should learn from achievable goals towards desired goals distribution progressively. To tackle the problem, we propose to generate a hindsight curriculum, which maintains a sequence of balancing distributions of achieved goals to replay. Based on the hindsight curriculum, the agent evaluates hindsight experiences with a batch of similar well-explored experiences, and strikes a dynamic balance between function approximation and task solving. We implement Hindsight Curriculum Generation (HCG) with the vanilla Deep Deterministic Policy Gradient (DDPG), and experiments on several multi-goal tasks with sparse binary rewards demonstrate that HCG improves sample efficiency of the state-of-the-art.