Dynamical phases of short-term memory mechanisms in RNNs

Bariscan Kurtkaya, Fatih Dinc, Mert Yuksekgonul, Marta Blanco-Pozo, Ege Cirakman, Mark Schnitzer, Yucel Yemez, Hidenori Tanaka, Peng Yuan, Nina Miolane

Abstract: Short-term memory is essential for cognitive processing, yet our understanding of its neural mechanisms remains unclear. Neuroscience has long focused on how sequential activity patterns, where neurons fire one after another within large networks, can explain how information is maintained. While recurrent connections were shown to drive sequential dynamics, a mechanistic understanding of this process still remains unknown. In this work, we introduce two unique mechanisms that can support this form of short-term memory: slow-point manifolds generating direct sequences or limit cycles providing temporally localized approximations. Using analytical models, we identify fundamental properties that govern the selection of each mechanism. Precisely, on short-term memory tasks (delayed cue-discrimination tasks), we derive theoretical scaling laws for critical learning rates as a function of the delay period length, beyond which no learning is possible. We empirically verify these results by training and evaluating approximately 80,000 recurrent neural networks (RNNs), which are publicly available for further analysis. Overall, our work provides new insights into short-term memory mechanisms and proposes experimentally testable predictions for systems neuroscience.

Lay Summary: How does the brain hold on to information for a few seconds, like remembering a phone number just long enough to dial it? This kind of short-term memory is critical for decision-making and everyday thinking, but how it works in the brain is still not fully understood. Earlier works have discovered that patterns of activity in the brain, where different neurons fire in a particular sequence, help maintain information over short timescales. However, the exact mechanisms behind how these sequences are formed and sustained remain unclear. In our study, we use simple mathematical models to show that short-term memory can be supported in two main ways: either through slow-moving activity patterns (like stepping through a series of states) or through repeating cycles (like a loop). We find clear mathematical rules that explain when each type of memory emerges and show that there's a limit to how long a system can hold information using these mechanisms. To test our theory, we trained nearly 80,000 artificial neural networks, which are made freely available to other researchers. Our work helps explain how short-term memory might work in both brains and machines, and we offer new predictions that can be tested in neuroscience experiments.

Link To Code: https://github.com/fatihdinc/dynamical-phases-stm

Primary Area: Applications->Neuroscience, Cognitive Science

Keywords: Dynamical system theory, computational neuroscience, working memory

Flagged For Ethics Review: true

Submission Number: 15002