Ubiquity of Hebbian Dynamics in Complex Learning Rules

David Aaron Koplow, Tomaso Poggio, Liu Ziyin

Keywords: Hebbian Learning, Machine learning, Neuroscience, Learning Dynamics

TL;DR: Hebbian learning emerges from L2 regularized learning rules as they approach stationarity (though before learning has ceased)

Abstract: Hebbian and anti-Hebbian plasticity are widely observed in the biological brain, yet their theoretical understanding remains limited. In this work, we find that when a learning method is regularized with L2 weight decay, its learning signal will gradually align with the direction of the Hebbian learning signal as it approaches stationarity. This Hebbian-like behavior is not unique to SGD: almost any learning rule, including random ones, can exhibit the same signature long before learning has ceased. We also provide a theoretical explanation for anti-Hebbian plasticity in regression tasks, demonstrating how it can arise naturally from gradient or input noise, and offering a potential reason for the observed anti-Hebbian effects in the brain. Certainly, our proposed mechanisms do not rule out any conventionally established forms of Hebbian plasticity and could coexist with them extensively in the brain. A key insight for neurophysiology is the need to develop ways to experimentally distinguish these two types of Hebbian observations.

Primary Area: applications to neuroscience & cognitive science

Submission Number: 20416