End-to-End Learning Framework for Solving Non-Markovian Optimal Control

Xiaole Zhang, Peiyu Zhang, Xiongye Xiao, Shixuan Li, Vasileios Tzoumas, Vijay Gupta, Paul Bogdan

TL;DR: A novel approach for the optimal control of fractional-order linear time-invariant (LTI) systems via the linear quadratic regulator (LQR)

Abstract: Integer-order calculus fails to capture the long-range dependence (LRD) and memory effects found in many complex systems. Fractional calculus addresses these gaps through fractional-order integrals and derivatives, but fractional-order dynamical systems pose substantial challenges in system identification and optimal control tasks. In this paper, we theoretically derive the optimal control via linear quadratic regulator (LQR) for fractional-order linear time-invariant (FOLTI) systems and develop an end-to-end deep learning framework based on this theoretical foundation. Our approach establishes a rigorous mathematical model, derives analytical solutions, and incorporates deep learning to achieve data-driven optimal control of FOLTI systems. Our key contributions include: (i) proposing a novel method for system identification and optimal control strategy in FOLTI systems, (ii) developing the first end-to-end data-driven learning framework, Fractional-Order Learning for Optimal Control (FOLOC), that learns control policies from observed trajectories, and (iii) deriving theoretical bounds on the sample complexity for learning accurate control policies under fractional-order dynamics. Experimental results indicate that our method accurately approximates fractional-order system behaviors without relying on Gaussian noise assumptions, pointing to promising avenues for advanced optimal control.

Lay Summary:

Markovian dynamics often fail to capture the long-range dependencies and memory effects present in many real-world processes. In contrast, fractional non-Markovian dynamics have shown promise as a powerful modeling tool for complex systems with memory. However, identifying such systems and designing optimal controllers remain challenging due to the analytical complexity of fractional calculus. To address this, we aim to provide both a theoretical convergence-guaranteed control algorithm and an end-to-end machine learning framework to simplify the control of fractional-order dynamical systems.

We develop a two-step algorithm that first performs system identification (i.e., parameter estimation) and then synthesizes an optimal controller. Alongside this, we provide sample complexity guarantees, quantifying the number of samples required to achieve accurate control in complex real-world settings. We also introduce Fractional-Order Learning for Optimal Control (FOLOC), the first end-to-end data-driven framework that learns control policies directly from observed trajectories.

Our contributions include a theoretically grounded learning algorithm and a practical machine learning framework for solving optimal control problems in fractional-order dynamical systems. This work offers a novel tool for learning and controlling non-Markovian systems, enabling more accurate modeling and control of real-world processes.

Link To Code: https://github.com/zpykillcc/Fractional-Order-Learning-for-Optimal-Control-Framework

Primary Area: Optimization

Keywords: Fractional-order dynamical system, Optimal control

Submission Number: 9590