LEARNING FROM LESS: SINDY SURROGATES IN RL
Keywords: reinforcement learning, surrogate models, sparse identification of nonlinear dynamics, SINDy, sample efficiency, model-based RL, OpenAI Gym, computational efficiency, interpretable models
TL;DR: SINDy-based surrogate environments accurately model reinforcement learning dynamics with minimal data, reducing computational costs while maintaining performance.
Abstract: This paper introduces a novel approach for developing surrogate environments in reinforcement learning (RL) using the Sparse Identification of Nonlinear Dynamics (SINDy) algorithm. We demonstrate the effectiveness of our approach through extensive experiments in OpenAI Gym environments, particularly Mountain Car and Lunar Lander. Our results show that SINDy-based surrogate models can accurately capture the underlying dynamics of these environments while reducing computational costs by 20-35%. By leveraging only 75 interactions for Mountain Car and 1000 for Lunar Lander, we achieve state-wise correlations exceeding 0.997, with mean squared errors as low as $3.11 \times 10^{-6}$ for Mountain Car velocity and $1.42 \times 10^{-6}$ for LunarLander position. RL agents trained in these surrogate environments require fewer total steps (65,075 vs. 100,000 for Mountain Car and 801,000 vs. 1,000,000 for Lunar Lander) while achieving comparable performance to those trained in the original environments, exhibiting similar convergence patterns and final performance metrics. This work contributes to the field of model-based RL by providing an efficient method for generating accurate, interpretable surrogate environments.
Submission Number: 81
Loading