Online Policy Optimization in Unknown Nonlinear Systems

Emile Timothy Anand, Yiheng Lin, James A Preiss, Fengze Xie, Soon-Jo Chung, Yisong Yue, Adam Wierman

Abstract: We study online policy optimization in nonlinear time-varying dynamical systems where the true dynamical models are unknown to the controller. This problem is challenging because, unlike in linear systems, the controller cannot obtain globally accurate estimations of the ground-truth dynamics using local exploration. We propose a meta-framework that combines a general online policy optimization algorithm (ALG) with a general online estimator of the dynamical system’s model parameters (EST). We show that if the hypothetical joint dynamics induced by ALG with known parameters satisfies several desired properties, the joint dynamics under inexact parameters from EST will be robust to errors. Importantly, the final policy regret only depends on EST’s predictions on the visited trajectory, which relaxes a bottleneck on identifying the true parameters globally. To demonstrate our framework, we develop a computationally efficient variant of Gradient-based Adaptive Policy Selection, called Memoryless GAPS (M-GAPS), and use it to instantiate ALG. Combining M-GAPS with online gradient descent to instantiate EST yields (to our knowledge) the first local regret bound for online policy optimization in nonlinear time-varying systems with unknown dynamics.