Understanding Memory in Neural Networks through Fisher Information Diffusion

ICLR 2026 Conference Submission13826 Authors

Keywords: Fisher information, information diffusion, initialization

Abstract: Information retention and transmission are fundamental to both artificial and biological neural networks. We present a general theoretical framework showing how information can be maintained on dynamically stable manifolds that evolve over time while preserving the geometry of inputs. In contrast to classical memory models such as Hopfield networks, which rely on static attractors, our approach highlights evolving stable subspaces as the substrate of memory. A central contribution of our work is the use of dynamic mean-field theory to uncover a new principle: operating at criticality (spectral radius $\approx 1$) is necessary but not sufficient for reliable information retention. Equally crucial—yet overlooked in prior studies—is the alignment between the input structure and the stable subspace. The theory leads to simple initialization rules that guarantee stable dynamics at the edge of chaos. We validate these rules in basic recurrent networks, showing that Fisher information–optimized initialization accelerates convergence and improves accuracy in sequential memory tasks, including the copy task and sequential MNIST compared to standard random initialization. Together, these results provide both principled design guidelines for recurrent networks and new theoretical insight into how information can be preserved over time.

Primary Area: learning theory

Submission Number: 13826