Learning by shaking: Computing policy gradients by physical forward-propagation

Arash Mehrjou, Ashkan Soleymani, Stefan Bauer, Bernhard Schölkopf

TL;DR: We propose a method to learn the effect of changing the parameters of the policy on the produced trajectories directly from the physical system.

Abstract: Model-free and model-based reinforcement learning are two ends of a spectrum. Learning a good policy without a dynamic model can be prohibitively expensive. Learning the dynamic model of a system can reduce the cost of learning the policy, but it can also introduce bias if it is not accurate. We propose a middle ground where instead of the transition model, the sensitivity of the trajectories with respect to the perturbation (shaking) of the parameters is learned. This allows us to predict the local behavior of the physical system around a set of nominal policies without knowing the actual model. We assay our method on a custom-built physical robot in extensive experiments and show the feasibility of the approach in practice. We investigate potential challenges when applying our method to physical systems and propose solutions to each of them.

Code: https://sites.google.com/view/physicalderivatives/

Keywords: Reinforcement Learning, Control Theory