Learning min-norm stabilizing control laws for systems with unknown dynamics

Tyler Westenbroek, Fernando Castaneda, Ayush Agrawal, Shankar Sastry, Koushil Sreenath

Abstract:

This paper introduces a framework for learning
a minimum-norm stabilizing controller for a system with unknown dynamics using model-free policy optimization methods.
The approach begins by first designing a Control Lyapunov
Function (CLF) for a (possibly inaccurate) dynamics model for
the system, along with a function which specifies a minimum
acceptable rate of energy dissipation for the CLF at different
points in the state-space. Treating the energy dissipation condition as a constraint on the desired closed-loop behavior of
the real-world system, we use penalty methods to formulate an
unconstrained optimization problem over the parameters of a
learned controller, which can be solved using model-free policy
optimization algorithms using data collected from the plant. We
discuss when the optimization learns a stabilizing controller for
the real world system and derive conditions on the structure
of the learned controller which ensure that the optimization is
strongly convex, meaning the globally optimal solution can be
found reliably. We validate the approach in simulation, first for
a double pendulum, and then generalize the framework to learn
stable walking controllers for underactuated bipedal robots
using the Hybrid Zero Dynamics framework. By encoding a
large amount of structure into the learning problem, we are
able to learn stabilizing controllers for both systems with only
minutes or even seconds of training data.