Hierarchical Reinforcement Learning with Hindsight

Andrew Levy, Robert Platt, Kate Saenko

Sep 27, 2018 ICLR 2019 Conference Blind Submission readers: everyone Show Bibtex
  • Abstract: We introduce a new Hierarchical Reinforcement Learning (HRL) framework that can accelerate learning in tasks involving long time horizons and sparse rewards. Our approach improves sample efficiency by enabling agents to simultaneously learn a hierarchy of short policies that operate at different time scales. These policy hierarchies can also support an arbitrary number of levels. Indeed, our framework is the first HRL approach to show results in which 3-level agents outperform both 2-level and 1-level agents in tasks with continuous state and action spaces. We demonstrate experimentally in both grid world and simulated robotics domains that our approach can significantly boost sample efficiency. A video illustrating our results is available at https://www.youtube.com/watch?v=i04QF7Yi50Y.
  • Keywords: Hierarchical Reinforcement Learning, Reinforcement Learning, Deep Reinforcement Learning
  • TL;DR: We introduce the first Hierarchical RL approach that has shown results in which 3 level agents outperform both 2-level and 1-level agents in continuous tasks.
0 Replies