Hierarchical Entity-centric Reinforcement Learning with Factored Subgoal Diffusion

ICLR 2026 Conference Submission14339 Authors

Keywords: Deep Reinforcement Learning, Goal-conditioned Reinforcement Learning, Object-centric Representations, Diffusion Subgoal Generation

TL;DR: We present a hierarchical entity-centric framework for offline goal-conditioned RL that produces entity-factored diffusion-generated subgoals for an RL agent, yielding consistent performance gains on long-horizon, image-based, sparse-reward tasks.

Abstract:

We propose a hierarchical entity-centric framework for offline Goal-Conditioned Reinforcement Learning (GCRL) that combines subgoal decomposition with factored structure to solve long-horizon tasks in domains with multiple entities.
Achieving long-horizon goals in complex environments remains a core challenge in Reinforcement Learning (RL). Domains with multiple entities are particularly difficult due to their combinatorial complexity. GCRL facilitates generalization across goals and the use of subgoal structure, but struggles with high-dimensional observations and combinatorial state-spaces, especially under sparse reward. We employ a two-level hierarchy composed of a value-based GCRL agent and a factored subgoal-generating conditional diffusion model. The RL agent and subgoal generator are trained independently and composed post hoc through selective subgoal generation based on the value function, making the approach modular and compatible with existing GCRL algorithms. We introduce new variations to benchmark tasks that highlight the challenges of entity-centric domains, and show that our method consistently boosts performance of the underlying RL agent on image-based long-horizon tasks with sparse rewards, achieving over $150$% higher success rates on the hardest task in our suite and generalizing to increasing horizons and numbers of entities. Rollout videos are provided at: https://sites.google.com/view/hecrl.

Primary Area: reinforcement learning

Submission Number: 14339