DiLQR: Differentiable Iterative Linear Quadratic Regulator via Implicit Differentiation

Shuyuan Wang, Philip D Loewen, Michael Forbes, Bhushan Gopaluni, Wei Pan

Abstract: While differentiable control has emerged as a powerful paradigm combining model-free flexibility with model-based efficiency, the iterative Linear Quadratic Regulator (iLQR) remains underexplored as a differentiable component. The scalability of differentiating through extended iterations and horizons poses significant challenges, hindering iLQR from being an effective differentiable controller. This paper introduces DiLQR, a framework that facilitates differentiation through iLQR, allowing it to serve as a trainable and differentiable module, either as or within a neural network. A novel aspect of this framework is the analytical solution that it provides for the gradient of an iLQR controller through implicit differentiation, which ensures a constant backward cost regardless of iteration, while producing an accurate gradient. We evaluate our framework on imitation tasks on famous control benchmarks. Our analytical method demonstrates superior computational performance, achieving up to $\textbf{128x}$ speedup and a minimum of $\textbf{21x}$ speedup compared to automatic differentiation. Our method also demonstrates superior learning performance ($\mathbf{10^6x}$) compared to traditional neural network policies and better model loss with differentiable controllers that lack exact analytical gradients. Furthermore, we integrate our module into a larger network with visual inputs to demonstrate the capacity of our method for high-dimensional, fully end-to-end tasks. Codes can be found on the project homepage~\url{https://sites.google.com/view/dilqr/}.

Lay Summary: Robots and autonomous systems often rely on controllers to make decisions, but combining these controllers with modern AI (like neural networks) is challenging because they aren’t designed to be "trainable" like other machine learning components. Our work, DiLQR, bridges this gap by making a powerful controller called iLQR compatible with AI training methods. Unlike standard approaches that are slow or inaccurate, DiLQR computes gradients (essential for training) efficiently and exactly, enabling up to 128x faster learning. We show that DiLQR outperforms both traditional controllers and neural networks in tasks like pendulum and cartpole control. This opens doors for more adaptable and efficient AI-driven control systems.

Primary Area: Reinforcement Learning->Planning

Keywords: iLQR; Differentiable control; Learning based control

Submission Number: 15377