Slow Transition to Low-Dimensional Chaos in Heavy-Tailed Recurrent Neural Networks

Yi Xie, Stefan Mihalas, Łukasz Kuśmierz

Keywords: theoretical neuroscience; recurrent neural networks; heavy-tailed connectivity; finite-size effects; Lyapunov exponents

TL;DR: We analytically predict the exact critical transition to chaos in finite-size heavy-tailed RNNs and reveal a tradeoff: their broader edge-of-chaos regime comes at the cost of lower attractor dimensionality compared to Gaussian networks.

Abstract: Growing evidence suggests that synaptic weights in the brain follow heavy-tailed distributions, yet most theoretical analyses of recurrent neural networks (RNNs) assume Gaussian connectivity. We systematically study the activity of RNNs with random weights drawn from biologically plausible Lévy alpha-stable distributions. While mean-field theory for the infinite system predicts that the quiescent state is always unstable---implying ubiquitous chaos---our finite-size analysis reveals a sharp transition between quiescent and chaotic dynamics. We theoretically predict the gain at which the finite system transitions from quiescent to chaotic dynamics, and validate it through simulations. Compared to Gaussian networks, finite heavy-tailed RNNs exhibit a broader gain regime near the edge of chaos, namely, a slow transition to chaos. However, this robustness comes with a tradeoff: heavier tails reduce the Lyapunov dimension of the attractor, indicating lower effective dimensionality. Our results reveal a biologically aligned tradeoff between the robustness of dynamics near the edge of chaos and the richness of high-dimensional neural activity. By analytically characterizing the transition point in finite-size networks---where mean-field theory breaks down---we provide a tractable framework for understanding dynamics in realistically sized, heavy-tailed neural circuits.

Primary Area: Neuroscience and cognitive science (e.g., neural coding, brain-computer interfaces)

Submission Number: 7446