How to train RNNs on chaotic data?

Zahra Monfared, Jonas Magdy Mikhaeil, Daniel Durstewitz

Keywords: dynamical systems, back-propagation through time, chaos, recurrent neural networks, LSTM, Lyapunov spectrum, time series

Abstract: Recurrent neural networks (RNNs) are wide-spread machine learning tools for modeling sequential and time series data. They are notoriously hard to train because their loss gradients backpropagated in time tend to saturate or diverge during training. This is known as the exploding and vanishing gradient problem. Previous solutions to this issue either built on rather complicated, purpose-engineered architectures with gated memory buffers, or - more recently - imposed constraints that ensure convergence to a fixed point or restrict (the eigenspectrum of) the recurrence matrix. Such constraints, however, convey severe limitations on the expressivity of the RNN. Essential intrinsic dynamics such as multistability or chaos are disabled. This is inherently at disaccord with the chaotic nature of many, if not most, time series encountered in nature and society. Here we offer a comprehensive theoretical treatment of this problem by relating the loss gradients during RNN training to the Lyapunov spectrum of RNN-generated orbits. We mathematically prove that RNNs producing stable equilibrium or cyclic behavior have bounded gradients, whereas the gradients of RNNs with chaotic dynamics always diverge. Based on these analyses and insights, we offer an effective yet simple training technique for chaotic data and guidance on how to choose relevant hyperparameters according to the Lyapunov spectrum.

One-sentence Summary: A mathematical framework that links RNN dynamics to the behavior of the loss gradients and offers a simple solution for training RNNs on chaotic data.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2110.07238/code)