Thermal Robustness of Retrieval in Dense Associative Memories: LSE vs LSR Kernels

Tatiana Petrova

Keywords: Modern Hopfield Networks, Associative Memory, Energy-Based Models, Exponential Capacity, Finite Temperature, Thermodynamic Stability, Monte Carlo Simulation

TL;DR: We map the finite-temperature retrieval stability boundary of modern Hopfield networks with exponential capacity (LSE and LSR) and demonstrate that finite-support LSR models exhibit significantly enhanced thermal robustness.

Abstract: Understanding whether retrieval in dense associative memories survives thermal noise is essential for bridging zero-temperature capacity proofs with the finite-temperature conditions of practical inference and biological computation. We use Monte Carlo simulations to map the retrieval phase boundary of two continuous dense associative memories (DAMs) on the $N$-sphere with an exponential number of stored patterns $M = e^{\alpha N}$: a log-sum-exp (LSE) kernel and a log-sum-ReLU (LSR) kernel. Both kernels share the zero-temperature critical load $\alpha_c(0)=0.5$, but their finite-temperature behavior differs markedly. The LSE kernel sustains retrieval at arbitrarily high temperatures for sufficiently low load, whereas the LSR kernel exhibits a finite support threshold below which retrieval is perfect at any temperature; for typical sharpness values this threshold approaches $\alpha_c$, making retrieval nearly perfect across the entire load range. We also compare the measured equilibrium alignment with analytical Boltzmann predictions within the retrieval basin.

Submission Number: 12