Reinforcement Learning for Control with Probabilistic Stability Guarantee

Minghao Han, Zhipeng Zhou, Lixian Zhang, Jun Wang, Wei Pan

Keywords: control, Lyapunov stability, REINFORCE, finite-sample bounds

Abstract: Reinforcement learning is promising to control dynamical systems for which the traditional control methods are hardly applicable. However, in control theory, the stability of a closed-loop system can be hardly guaranteed using the policy/controller learned solely from samples. In this paper, we will combine Lyapunov's method in control theory and stochastic analysis to analyze the mean square stability of MDP in a model-free manner. Furthermore, the finite sample bounds on the probability of stability are derived as a function of the number M and length T of the sampled trajectories. And we show that there is a lower bound on T and the probability is much more demanding for M than T. Based on the theoretical results, a REINFORCE like algorithm is proposed to learn the controller and the Lyapunov function simultaneously.

One-sentence Summary: Sample-based stability condition and the associated finite sample bound for reinforcement learning control.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=mUG11pz4m