Should EBMs model the energy or the score?

Tim Salimans, Jonathan Ho

Abstract: Recent progress in training unnormalized models through denoising score matching with Langevin dynamics (SMLD) and denoising diffusion probabilistic modeling (DDPM) has made unnormalized models a competitive model class for generative modeling. Unlike earlier work on energy based models, these recent works construct generative models by directly parameterizing the score function of the model density, rather than the density itself. Such unconstrained score models are not guaranteed to output a conservative vector field, meaning they do not correspond to the gradient of any function, unlike constrained score models that are obtained through explicitly differentiating a parameterized energy function. Explicit energy based models thus seem to have a theoretical advantage, but empirical evidence currently points to unconstrained score models performing better in practice. Here we compare both methods for modeling the score of the data distribution, finding that constrained score models, i.e. energy based models, can perform just as well as unconstrained models when using a comparable model structure.