Do recent advancements in model-based deep reinforcement learning really improve data efficiency?

Kacper Piotr Kielak

Keywords: deep learning, reinforcement learning, data efficiency, DQN, Rainbow, SimPLe

TL;DR: Recent advancements in data-efficient model-based reinforcement learning are not any more data efficient than existing model-free approaches.

Abstract: Reinforcement learning (RL) has seen great advancements in the past few years. Nevertheless, the consensus among the RL community is that currently used model-free methods, despite all their benefits, suffer from extreme data inefficiency. To circumvent this problem, novel model-based approaches were introduced that often claim to be much more efficient than their model-free counterparts. In this paper, however, we demonstrate that the state-of-the-art model-free Rainbow DQN algorithm can be trained using a much smaller number of samples than it is commonly reported. By simply allowing the algorithm to execute network updates more frequently we manage to reach similar or better results than existing model-based techniques, at a fraction of complexity and computational costs. Furthermore, based on the outcomes of the study, we argue that the agent similar to the modified Rainbow DQN that is presented in this paper should be used as a baseline for any future work aimed at improving sample efficiency of deep reinforcement learning.