Energy Minimization for Training Dense Associative Memory

Bao Pham, Benjamin Hoover, Dmitry Krotov, Parikshit Ram

Keywords: associative memory, energy minimization, energy-based learning, sampling-free

TL;DR: DenseAMs with translation-invariant similarity metrics can be trained using strict energy minimization, which is equivalent to MLE. We show how to learn quality memories in ambient (data) space and latent space

Abstract: Dense Associative Memories (DenseAMs) are modern generalizations of Hopfield networks with high-capacity, energy-based retrieval dynamics, but it remains unclear what the most elegant training principle should be for these models. Contrastive divergence (CD) is theoretically well motivated but requires expensive iterative negative sampling, and backpropagating reconstruction loss through long inference trajectories is also costly while not directly leveraging the explicit energy objective. Inspired by the Hebbian learning rule in classical Hopfield networks, we propose to train DenseAMs by direct energy minimization. For DenseAMs with translation-invariant kernel energies, we show that the partition function is independent of memory parameters, so maximum likelihood estimation (MLE) reduces exactly to minimizing data energy. This yields a sampling-free training rule that preserves an explicit energy formulation. We demonstrate the method in both ambient space and latent space, where a stop-gradient coupling with an autoencoder enables stable joint training and memory synthesis from latent noise.

Submission Number: 45