Realizable Abstractions: Near-Optimal Hierarchical Reinforcement Learning

Roberto Cipollone, Luca Iocchi, Matteo Leonetti

Keywords: Hierarchical Reinforcement Learning, Reinforcement Learning theory, PAC algorithm, MDP abstractions

TL;DR: The paper describes Realizable Abstractions, a new notion in Hierarchical Reinforcement Learning (HRL), and a novel algorithm RARL to find low-level policies in HRL that is proven to be Probably Approximately Correct.

Abstract:

The main focus of Hierarchical Reinforcement Learning (HRL) is studying how large Markov Decision Processes (MDPs) can be more efficiently solved when addressed in a modular way, by combining partial solutions computed for smaller subtasks.
Despite their very intuitive role for learning, most notions of MDP abstractions proposed in the HRL literature have limited expressive power or do not possess formal efficiency guarantees.

This work addresses these fundamental issues by defining Realizable Abstractions, a new relation between generic low-level MDPs and their associated high-level decision processes.
The notion we propose avoids non-Markovianity issues and has desirable near-optimality guarantees.
Indeed, we show that any abstract policy for Realizable Abstractions can be translated into near-optimal policies for the low-level MDP, through a suitable composition of options.
As demonstrated in the paper, these options can be expressed as solutions of specific constrained MDPs.
Based on these findings, we propose RARL, a new HRL algorithm that returns compositional and near-optimal low-level policies, taking advantage of the Realizable Abstraction given in the input.
We show that RARL is Probably Approximately Correct, it converges in a polynomial number of samples, and it is robust to inaccuracies in the abstraction.

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 10473