Chaos is a property of many dynamical systems. The impact of this phenomenon of chaos on recurrent neural networks (RNNs) is studied based on prior work. Some problems with the most common RNN architectures due to chaos and instability are analyzed theoretically and empirically. These are related to having strange attractors and the vanishing/exploding gradient problems that make it difficult to learn long-term dependencies. Novel architectures from the literature such as Chaos free RNN and Antisymmetric RNN claim to help mitigate these issues and ensure that RNNs learn good attractors. Some empirical analysis is carried out to study the advantages brought by these architectures. This is a learning attempt to view RNNs through the lens of dynamical systems and understand them in perspective. 

Université de Montréal Winter2021 MAT6215 Submissions

Recurrent Neural Networks as Dynamical Systems: Chaos and Order