Subgoal Reachability in Goal Conditioned Hierarchical Reinforcement Learning

Michal Bortkiewicz, Jakub Lyskawa, Pawel Wawrzynski, Mateusz Ostaszewski, Artur Grudkowski, Bartlomiej Sobieski, Tomasz Trzcinski

Abstract: Achieving long-term goals becomes more feasible when we break them into smaller, manageable subgoals. Yet, a crucial question arises: how specific should these subgoals be? Existing Goal-Conditioned Hierarchical Reinforcement Learning methods are based on lower-level policies aiming at subgoals designated by higher-level policies. These methods are sensitive to the proximity threshold under which the subgoals are considered achieved. Constant thresholds make the subgoals impossible to achieve in the early learning stages, easy to achieve in the late stages, and require careful manual tuning to yield reasonable overall learning performance. We argue that subgoal precision should depend on the agent’s recent performance rather than be predefined. We propose Adaptive Subgoal Required Distance (ASRD), a drop-in replacement method for subgoal threshold creation that considers the agent’s current lower-level capabilities for appropriate subgoals. Our results demonstrate that subgoal precis