Recurrent Neural Networks as Dynamical Systems: Chaos and Order

Pravish Sainath

Keywords: RNN, chaos, stability, long-term dependency, attractor

TL;DR: Analysis on the effect of chaos in RNNs and ways to bring in order.

Abstract: Chaos is a property of many dynamical systems. The impact of this phenomenon of chaos on recurrent neural networks (RNNs) is studied based on prior work. Some problems with the most common RNN architectures due to chaos and instability are analyzed theoretically and empirically. These are related to having strange attractors and the vanishing/exploding gradient problems that make it difficult to learn long-term dependencies. Novel architectures from the literature such as Chaos free RNN and Antisymmetric RNN claim to help mitigate these issues and ensure that RNNs learn good attractors. Some empirical analysis is carried out to study the advantages brought by these architectures. This is a learning attempt to view RNNs through the lens of dynamical systems and understand them in perspective.