Residual Model-Based Reinforcement Learning for Physical Dynamics

Zakariae EL ASRI, Clément Rambour, Vincent LE GUEN, Nicolas THOME

Keywords: Model-Based RL, ODE, Physical Dynamics, Residual Model

TL;DR: This paper presents a physical model-based RL framewrok with a data-driven residual able to generalize to complex environments.

Abstract: Dynamic control problems are a prevalent topic in robotics. Deep neural networks have been shown to learn accurately many complex dynamics, but these approaches remain data-inefficient or intractable in some tasks. Rather than learning to reproduce the environment dynamics, traditional control approaches use some physical knowledge to describe the environment's evolution. These approaches do not need many samples to be tuned but suffer from approximations and are not adapted to strong modifications of the environment. In this paper, we introduce a method to learn the parameters of a physical model \ie the parameter of an Ordinary Differential Equation (ODE) to approach at best the observed transitions. This model is completed with a residual data-driven term in charge to reduce the reality gap between simple physical priors and complex environments. We also show that this approach can be naturally extended to the case of the fine-tuning of an implicit physical model trained on simple simulations.