Hitchhiker's guide on the relation of Energy-Based Models with other generative models, sampling and statistical physics: a comprehensive review

Davide Carbone

Abstract: Energy-Based Models (EBMs) have emerged as a powerful framework in the realm of generative modeling, offering a unique perspective that aligns closely with principles of statistical mechanics. This review aims to provide physicists with a comprehensive understanding of EBMs, delineating their connection to other generative models such as Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and Normalizing Flows. We explore the sampling techniques crucial for EBMs, including Markov Chain Monte Carlo (MCMC) methods, and draw parallels between EBM concepts and statistical mechanics, highlighting the significance of energy functions and partition functions. Furthermore, we delve into state-of-the-art training methodologies for EBMs, covering recent advancements and their implications for enhanced model performance and efficiency. This review is designed to clarify the often complex interconnections between these models, which can be challenging due to the diverse communities working on the topic.

Submission Length: Long submission (more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=hbLKVxMtK2&noteId=hbLKVxMtK2

Changes Since Last Submission:

We thank the Action Editor and reviewers for their thoughtful and constructive feedback. We have carefully revised the manuscript and addressed all major concerns raised. Below, we summarize the changes made and provide context for our decisions. All modifications are highlighted in red in the manuscript.

- Clarification of the scope and objective of the review

We have rewritten the beginning of the introduction to clearly articulate the aim of the review. We also included comparisons with existing surveys on EBMs, such as [LeCun et al., 2006], [Song & Ermon, 2021], [Bond et al., 2021], and [Huembeli et al., 2022]. Our work is not meant to duplicate those efforts, but to provide a **conceptual guide** that situates EBMs at the intersection of *sampling, statistical physics, and generative modeling*. We believe this integrative perspective is currently lacking in the literature.
As requested, the bullet-point roadmap was moved into its own paragraph and is now properly contextualized.

- Historical perspective moved to the appendix

In response to multiple comments regarding the length of the historical overview, we have moved this content to an appendix. In its place, we provide a brief summary and focus on motivating the central narrative. We now emphasize our key message early on: *we take the viewpoint that EBMs should not be understood solely as a machine learning model class defined by an unnormalized density function, but rather as a natural interface between statistical physics, sampling theory, and modern generative modeling.*

- Merging and shortening Sections 3 and 4

We have merged the two sections into a single one discussing the relation of EBMs to other generative models. The treatments of VAEs and GANs have been shortened, and the section on stochastic interpolants has been moved to the appendix.

However, we believe that removing other sections of this material would reduce the accessibility of the review for readers who are not already familiar with these models. Our intention is to keep the review **self-contained and pedagogical**, and we think this level of detail is necessary to support interdisciplinary readership. The revised section occupies ~10 out of 32 pages, which we consider a reasonable compromise.

- Expansion of Section 7 (now Section 6) on recent EBM training

We expanded the section significantly, renamed it **"Recent developments in EBM training"**, and included coverage of score-based methods, modern Langevin approaches, and other recent techniques. The term "state-of-the-art" has been revised as appropriate.

- Final section on open questions and future directions

We added a dedicated subsection in the conclusions outlining future research directions and open problems, with emphasis on the role of statistical physics in inspiring new learning and sampling strategies. We hope this can foster further cross-disciplinary engagement.

---

## Final remarks

We respectfully emphasize that this review is not intended to be *only* a survey of EBMs or *only* a tutorial on generative modeling. Rather, it aims to serve as a **conceptual guide** through the interrelations between EBMs, sampling, physics, and generative paradigms. We feel that this specific focus is not yet covered by existing reviews, and we hope it offers value to both ML and physics audiences.

As we wrote in our previous response:
> *“Our hope is that this review will serve as both a useful reference and a source of new perspectives. If, after reading it, a reader were to think, ‘This is clear, this as well… but this connection—I had never considered it before,’ we would consider it a great success.”*

We thank you again for your careful consideration, and we hope the revised version aligns more closely with the standards of TMLR.

We removed the parts in red.

Assigned Action Editor: ~Alberto_Bietti1

Submission Number: 3584