Memory in Plain Sight: A Survey of the Uncanny Resemblances between Diffusion Models and Associative Memories

Benjamin Hoover, Hendrik Strobelt, Dmitry Krotov, Judy Hoffman, Zsolt Kira, Duen Horng Chau

Keywords: Associative Memory, lyapunov, energy, diffusion, denoising, ODE, Hopfield Networks

TL;DR: This submission describes a here-to-fore unseen equivalence between the reverse (denoising) process of Diffusion Models and the memory retrieval process of Associative Memories.

Abstract: Diffusion Models (DMs) have recently set state-of-the-art on many generation benchmarks. However, there are myriad ways to describe them mathematically, which makes it difficult to develop a simple understanding of how they work. In this submission, we provide a concise overview of DMs from the perspective of dynamical systems and Ordinary Differential Equations (ODEs) which exposes a mathematical connection to the highly related yet often overlooked class of energy-based models, called Associative Memories (AMs). Energy-based AMs are a theoretical framework that behave much like denoising DMs, but they enable us to directly compute a Lyapunov energy function on which we can perform gradient descent to denoise data. We finally identify the similarities and differences between AMs and DMs, discussing new research directions revealed by the extent of their similarities.

Submission Number: 15