A Hierarchical Goal-Biased Curriculum for Training Reinforcement Learning

Sunandita Patra, Mark Cavolowsky, Onur Kulaksizoglu, Ruoxi Li, Laura Hiatt, Mark Roberts, Dana S. Nau

Abstract: Hierarchy and curricula are two techniques commonly used to improve training for Reinforcement Learning (RL) agents. Yet few works have examined how to leverage hierarchical planning to generate a curriculum for training RL Options. We formalize a goal skill that extends an RL Option with state-based conditions that must hold during training and execution. We then define a Goal-Skill Network that integrates a Hierarchical Goal Network, a variant of hierarchical planning, with goal skills as the leaves of the network. An automatically generated plan for a Goal-Skill Network correctly orders goal skills such that (1) it is a Goal-Biased Curriculum for training the goal skills, and (2) it can be executed to achieve top-level goals. In a set of six distinct gridworld environments using up to ten goal skills, we demonstrate that these contributions train nearly perfect policies significantly faster than learning a whole policy from scratch.