Model-Based Reinforcement Learning with SINDy

Rushiv Arora, Eliot Moss, Bruno Castro da Silva

Keywords: model-based Reinforcement Learning, model learning, non-linear dynamical systems

TL;DR: It is easy to extract models from physics based systems and use them to learn (near-)optimal policies quickly from small(er) amounts of data.

Abstract:

We draw on the latest advancements in the physics community to propose a novel
method for discovering the governing non-linear dynamics of physical systems
in reinforcement learning (RL). We establish that this method is capable of
discovering the underlying dynamics using significantly fewer trajectories (as
little as one rollout with $\leq 30$ time steps) than state of the art model
learning algorithms. Further, the technique learns a model that is accurate
enough to induce near-optimal policies given significantly fewer trajectories
than those required by model-free algorithms. It brings the benefits of
model-based RL without requiring a model to be developed in advance, for
systems that have physics-based dynamics.

To establish the validity and applicability of this algorithm, we conduct
experiments on four classic control tasks. We found that an optimal policy
trained on the discovered dynamics of the underlying system can generalize
well. Further, the learned policy performs well when deployed on the actual
physical system, thus bridging the model to real system gap. We further
compare our method to state-of-the-art model-based and model-free approaches,
and show that our method requires fewer trajectories sampled on the true
physical system compared other methods. Additionally, we explored approximate
dynamics models and found that they also can perform well.