Exploring by Exploiting Bad Models in Model-Based Reinforcement Learning

Yixin Lin, Sarah Bechtle, Ludovic Righetti, Akshara Rai, Franziska Meier

Abstract:

Exploration for reinforcement learning (RL) is well-studied for model-free methods but a relatively unexplored topic for model-based methods.
In this work, we investigate several exploration techniques injected into the two stages of model-based RL:
(1) during optimization: adding transition-space and action-space noise when optimizing a policy using learned dynamics, and
(2) after optimization: injecting action-space noise when executing an optimized policy on the real environment.
When given a good deterministic dynamics model, like the ground-truth simulation, exploration can significantly improve performance.
However, using randomly initialized neural networks to model environment dynamics can _implicitly_ induce exploration in model-based RL, reducing the need for explicit exploratory techniques.
Surprisingly, we show that in the case of a local optimizer, using a learned model with this implicit exploration can actually _outperform_ using the ground-truth model without exploration, while adding exploration to the ground-truth model reduces the performance gap. However, the learned models are highly local, in that they perform well _only_ for the task for which it is optimized, and fail to generalize to new targets.