Breadth First Exploration in Grid-based Reinforcement Learning

Youngsik Yoon, Gangbok Lee, Sungsoo Ahn, Jungseul Ok

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: goal-conditioned RL, graph-based RL

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

Recently, graph-based planners have gained significant attention for goal-conditioned reinforcement learning (RL), where they construct a graph that represents confident transitions between "subgoals'' as edges and run shortest path algorithms to exploit the confident edges. Such a graph construction consists of only achieved subgoals while recording unattained ones in history is also crucial. Indeed, it often wastes an excessive number of attempts to unattainable subgoals.
To alleviate this issue, we propose a graph construction method that efficiently manages all the achieved and unattained subgoals on a grid graph adaptively discretizing the goal space. This enables a breadth-first exploration strategy, grounded in local adaptive grid refinement,
that prioritizes broad probing of subgoals on a coarse grid over meticulous one on a dense grid. We empirically verify the effectiveness of our method through extensive experiments.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3445