Model-based reinforcement learning with nearly tight exploration complexity bounds

Istvan Szita, Csaba Szepesvári

Abstract: One might believe that model-based algorithms of reinforcement learning can propagate the obtained experience more quickly, and are able to direct exploration better. As a consequence, fewer exploratory actions should be enough to learn a good policy. Strangely enough, current theoretical results for model-based algorithms do not support this claim: In a nite Markov decision process with N states, the best bounds on the number of exploratory steps necessary are of order O(N 2 logN), in contrast to the O(N logN) bound available for the modelfree, delayed Q-learning algorithm. In this paper we show that Mormax, a modied version of the Rmax algorithm needs to make at most O(N logN) exploratory steps. This matches the lower bound up to logarithmic factors, as well as the upper bound of the state-of-the-art model-free algorithm, while our new bound improves the dependence on other problem parameters.