Online abstraction with MDP homomorphisms for Deep Learning

Ondrej Biza, Robert Platt

Abstract: Abstraction of Markov Decision Processes is a useful tool for solving complex problems, as it can ignore unimportant aspects of an environment, simplifying the process of learning an optimal policy. In this paper, we propose a new algorithm for finding abstract MDPs in environments with continuous state spaces. It is based on MDP homomorphisms, a structure-preserving mapping between MDPs. We demonstrate our algorithm's ability to learns abstractions from collected experience and show how to reuse the abstractions to guide exploration in new tasks the agent encounters. Our novel task transfer method beats a baseline based on a deep Q-network.

Keywords: reinforcement learning, abstraction, mdp homomorphism, deep learning, robotics

TL;DR: We create abstract models of environments from experience and use them to learn new tasks faster.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:1811.12929/code)