Reinforcement Learning vs Optimal Control: Sparse Nonlinear Dynamical Systems Between Theory and Practice

Davide Maran, Gianmarco Tedeschi, Enea Gusmeroli, Marcello Restelli

Keywords: Reinforcement Learning, Optimal Control, Sparsity, Sindy, No-regret

Abstract: The recent development of sparse methods for identifying nonlinear dynamical systems has opened new avenues for efficient and interpretable model-based reinforcement learning (RL). In this work, we study online RL in environments where the system dynamics, modeled as $s'=f(s,a)+$noise, is assumed to be sparse with respect to a big feature map, a structural idea inspired by the SINDy framework. We introduce an optimistic algorithm that combines online sparse regression with confidence set construction to guide exploration and planning. Our theoretical contributions are threefold: (i) we provide the first regret bounds for sparse nonlinear dynamics, showing that regret scales with the sparsity level $d_0$; (ii) we relax standard Gaussian assumptions by allowing general subgaussian noise with bounded variation densities, significantly broadening the class of admissible stochastic systems; and (iii) we extend our theoretical guarantees to misspecified models, where the dynamics are only approximately sparse in the chosen feature space. The algorithm enjoying the regret bound is not computationally efficient, as it builds on a very heavy online regression method. We propose a practical variant using ensemble SINDy in place of the online regression algorithm, and SAC within a Dyna-style framework. Empirical results on classic continuous control tasks demonstrate the practical viability and robustness of our approach.

Confirmation: I understand that authors of each paper submitted to EWRL may be asked to review 2-3 other submissions to EWRL.

Serve As Reviewer: ~Davide_Maran1, ~Gianmarco_Tedeschi1

Track: Regular Track: unpublished work

Submission Number: 103