Learning Context-Sensitive State and Action Abstractions for Reinforcement Learning with Parameterized Actions

Rashmeet Kaur Nayyar, Naman Shah, Siddharth Srivastava

Keywords: Parameterized/Hybrid Actions, Learning State and Action Abstractions

TL;DR: A novel domain-independent RL approach that addresses problems with parameterized action spaces in long-horizon, sparse-reward settings.

Abstract: Real-world sequential decision making problems often require parameterized action spaces that feature both, decisions regarding discrete actions and decisions about continuous action parameters governing how an action is executed. However, existing approaches exhibit severe limitations when handling such parameterized action spaces---planning algorithms require hand-crafted action models, and reinforcement learning (RL) paradigms focus on either discrete or continuous actions but not both. This paper extends the scope of RL algorithms to settings with mixtures of discrete and continuous parameterized actions through a unified view of continuous-to-discrete context-sensitive state and action abstractions. We present algorithms for online learning and flexible refinement of such abstractions during RL. Empirical results show that learning such abstractions on-the-fly enable Q-learning to significantly outperform state-of-the-art RL approaches in terms of sample efficiency across diverse problem domains with long horizons, continuous states and parameterized actions.

Submission Number: 22