How Chaotic Are Recurrent Neural Networks?

Pourya Vakilipourtakalou, Lili Mou

Keywords: RNN, Chaos, NLP, Text Generation

TL;DR: We analyze the chaotic behavior of Recurrent Neural Networks (RNN) and find that in real-world applications, e.g., text generation, RNNs do not perform chaotic behavior.

Abstract:

Recurrent neural networks (RNNs) are non-linear dynamic systems. Previous
work believes that RNN may suffer from the phenomenon of chaos, where the
system is sensitive to initial states and unpredictable in the long run. In this paper,
however, we perform a systematic empirical analysis, showing that a vanilla or
long short term memory (LSTM) RNN does not exhibit chaotic behavior along
the training process in real applications such as text generation. Our findings suggest
that future work in this direction should address the other side of non-linear
dynamics for RNN.