Why do recurrent neural networks suddenly learn? Bifurcation mechanisms in neuro-inspired short-term memory tasks

Udith Haputhanthri, Liam Storan, Yiqi Jiang, Adam Shai, Hakki Orhun Akengin, Mark Schnitzer, Fatih Dinc, Hidenori Tanaka

Keywords: neuroscience, recurrent neural networks, bifurcations

Abstract: Recurrent neural networks (RNNs) are regularly studied as in silico models of biological and artificial computation. Training RNNs requires updating many synaptic weights, making the learning process complex and high-dimensional. In order to uncover learning mechanisms, we investigated the sudden accuracy jumps in RNNs' loss curves. Across several short-term memory tasks, we identified an initial search phase with accuracy plateaus, followed by rapid acquisition of skills. Studying attractor landscapes during learning revealed high-dimensional bifurcations as the links between these phases. Next, we introduced the temporal consistency regularization (TCR), a biologically plausible learning rule that incentivizes formation of memory-subserving attractors. In diverse short-term memory tasks, TCR accelerated (online) training, promoted robust attractors, and enabled networks initialized in a chaotic regime to train efficiently. Our analyses lead to testable predictions for system neuroscientists and highlight the need to study high-dimensional dynamical system theory to uncover learning mechanisms in biological and artificial networks.

Submission Number: 57