Go to EWRL 2024 Workshop homepageState Abstraction Discovery from Progressive Disaggregation Methods
The high dimensionality of model-based Reinforcement Learning (RL) and Markov Decision Processes (MDPs) can be reduced using abstractions of the state and action spaces. Although hierarchical learning and state abstraction methods have been explored over the past decades, explicit methods to build useful abstractions of models are rarely provided. In this work, we study the relationship between Approximate Dynamic Programming (ADP) and State Abstraction. We provide an estimation of the approximation made through abstraction, which can be explicitly calculated. We also introduce a way to solve large MDPs through an abstraction refinement process that can be applied to both discounted and total reward criteria. This method allows finding explicit state abstractions while solving any MDP with controlled error. We then integrate this state space disaggregation process into classical Dynamic Programming algorithms, namely Approximate Value Iteration, Q-Value Iteration, and Policy Iteration. We show that this method can decrease the solving time of a wide range of models and can also describe the underlying dynamics of the MDP without making any assumptions about the structure of the problem. We also conduct an extensive numerical comparison and compare our approach to existing aggregation methods to support our claims.