Self-supervised consolidation of latent dynamics for training recurrent neural networks at scale

NeurIPS 2025 Workshop NeurReps Submission117 Authors

Keywords: recurrent neural networks, computational neuroscience

Abstract: Recurrent neural networks are often used as model organisms to study how populations of neurons perform behavioral tasks. While robustness in biology is typically linked to large networks with millions of neurons, training such networks is slow and inefficient. Most computational studies therefore rely on much smaller models, limiting insights into scalability of learned representations. Here, we discuss a self-supervised framework in which a small, fast-learning part of the recurrent network is trained with backpropagation-through-time. Once trained, its latent dynamics are then consolidated into a much larger portion of the network, and then, the latter acquires task-specific input–output mappings within tens of training epochs. This two-step process parallels biological consolidation, where rapid learning in small circuits guides large-scale network organization. Using this approach, large recurrent networks develop stable, robust representations from sparse training signals, providing a biologically inspired path toward training scalable models of neural computation.

Submission Number: 117