Dense Associative Memory with Epanechnikov energy

Benjamin Hoover, Krishna Balasubramanian, Dmitry Krotov, Parikshit Ram

Track: long paper (up to 5 pages)

Keywords: Associative Memory, Dense Associative Memory, Kernel Density Estimation, Epanechnikov kernel, energy-based models, exponential memory capacity, Hopfield Network, spurious minima

TL;DR: We propose a Dense Associative Memory using a ReLU-based energy that achieves perfect memory retrieval and creative generation while maintaining exponential storage capacity, a combination previously thought impossible without exponentials

Abstract: We propose a novel energy function for Dense Associative Memory (DenseAM) networks, the log-sum-ReLU (LSR), inspired by optimal kernel density estimation. Unlike the common log-sum-exponential (LSE) function, LSR is based on the Epanechnikov kernel and enables exact memory retrieval with exponential capacity without requiring exponential separation functions. Uniquely, it introduces abundant additional emergent local minima while preserving perfect pattern recovery--a characteristic previously unseen in DenseAM literature. Empirical results show LSR generates significantly more local minima and produces samples with higher log-likelihood than LSE-based models, making it promising for both memory storage and generative tasks.

Submission Number: 25