Multi-step Greedy Policies in Model-Free Deep Reinforcement Learning

Yonathan Efroni, Manan Tomar, Mohammad Ghavamzadeh

Keywords: Reinforcement Learning, Multi-step greedy policies, Model free Reinforcement Learning

TL;DR: Use model free algorithms like DQN/TRPO to solve short horizon problems (model free) iteratively in a Policy/Value Iteration fashion.

Abstract: Multi-step greedy policies have been extensively used in model-based Reinforcement Learning (RL) and in the case when a model of the environment is available (e.g., in the game of Go). In this work, we explore the benefits of multi-step greedy policies in model-free RL when employed in the framework of multi-step Dynamic Programming (DP): multi-step Policy and Value Iteration. These algorithms iteratively solve short-horizon decision problems and converge to the optimal solution of the original one. By using model-free algorithms as solvers of the short-horizon problems we derive fully model-free algorithms which are instances of the multi-step DP framework. As model-free algorithms are prone to instabilities w.r.t. the decision problem horizon, this simple approach can help in mitigating these instabilities and results in an improved model-free algorithms. We test this approach and show results on both discrete and continuous control problems.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:1910.02919/code)