Dense Associative Memory with Epanechnikov Energy

Benjamin Hoover, Zhaoyang Shi, Krishna Balasubramanian, Dmitry Krotov, Parikshit Ram

Keywords: Associative Memory, Dense Associative Memory, Kernel Density Estimation, Epanechnikov kernel, energy-based models, exponential memory capacity, Hopfield Network, spurious minima

TL;DR: Changing the DenseAM kernel from the standard Gaussian kernel to the KDE-optimal Epanechnikov kernel results in 1) exponential capacity without the exponential and 2) meaningful, emergent memories.

Abstract: We propose a novel energy function for Dense Associative Memory (DenseAM) networks, the log-sum-ReLU (LSR), inspired by optimal kernel density estimation. Unlike the common log-sum-exponential (LSE) function, LSR is based on the Epanechnikov kernel and enables exact memory retrieval with exponential capacity without requiring exponential separation functions. Uniquely, it introduces abundant additional emergent local minima while preserving perfect pattern recovery --- a characteristic previously unseen in DenseAM literature. Empirical results show that LSR energy has significantly more local minima (memories) that have comparable log-likelihood to LSE-based models. Analysis of LSR's emergent memories on image datasets reveals a degree of creativity and novelty, hinting at this method's potential for both large-scale memory storage and generative tasks.

Primary Area: General machine learning (supervised, unsupervised, online, active, etc.)

Submission Number: 18013