Ubiquity of Hebbian Dynamics in Complex Learning Rules

ICLR 2026 Conference Submission20416 Authors

Keywords: Hebbian Learning, Machine learning, Neuroscience, Learning Dynamics

TL;DR: Hebbian learning emerges from L2 regularized learning rules as they approach stationarity (though before learning has ceased)

Abstract: Hebbian learning is widely regarded as a core principle of synaptic plasticity in the brain, yet it appears fundamentally different from Stochastic Gradient Descent (SGD), the optimization method that underlies nearly all modern artificial intelligence. We find that when a neural network is regularized with L2 weight decay, the gradient of the loss aligns with the direction of the Hebbian learning signal as it approaches stationarity. Surprisingly, this Hebbian-like behavior is not unique to SGD: almost any learning rule, including random ones, can exhibit the same signature long before learning has ceased. We also provide a theoretical explanation for anti-Hebbian plasticity in regression tasks, showing how it can arise naturally from gradient or input noise, offering a potential mechanism for anti-Hebbian observations in the brain. These results may help bridge a long-standing gap between artificial and biological learning, revealing Hebbian properties as an epiphenomenon of deeper optimization principles. Although there is evidence that the brain employs dedicated Hebbian mechanisms, much of the Hebbian-like plasticity observed may instead arise from more general optimization processes. From a neuroscience perspective, this suggests that Hebbian plasticity could be less central to learning than traditionally assumed. For machine learning, it highlights a surprising universality across algorithms that could inspire more efficient learning methods.

Primary Area: applications to neuroscience & cognitive science

Submission Number: 20416