An Analysis of Model-Based Reinforcement Learning From Abstracted Observations
Abstract: Many methods for Model-based Reinforcement learning (MBRL) in Markov decision processes (MDPs) provide guarantees for both the accuracy of the model they can deliver and the learning efficiency. At the same time, state abstraction techniques allow for a reduction of the size of an MDP while maintaining a bounded loss with respect to the original problem. Therefore, it may come as a surprise that no such guarantees are available when combining both techniques, i.e., where MBRL merely observes abstract states. Our theoretical analysis shows that abstraction can introduce a dependence between samples collected online (e.g., in the real world). That means that, without taking this dependence into
account, results for MBRL do not directly extend to this setting. Our result shows that we can use concentration inequalities for martingales to overcome this problem. This result makes it possible to extend the guarantees of existing MBRL algorithms to the setting with abstraction. We illustrate this by combining R-MAX, a prototypical MBRL algorithm, with abstraction, thus producing the first performance guarantees for model-based ‘RL from Abstracted Observations’: model-based reinforcement learning with an abstract model.
Submission Length: Long submission (more than 12 pages of main content)
Changes Since Last Submission: Minor revisions in the introduction, Section 3.4, and the conclusion.
The changes:
* In the middle of the introduction, highlight the comparison between learning with and without abstraction.
* At the end of the introduction, highlight the importance of the results.
* At the end of Section 3.4, added clarification on why there is no fixed set of weights.
* At the end of the conclusion, clarification on the importance of the results.
Assigned Action Editor: ~Naman_Agarwal1
License: Creative Commons Attribution 4.0 International (CC BY 4.0)
Submission Number: 796
Loading