Generative Marginalization Models

Sulin Liu, Peter Ramadge, Ryan P Adams

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: discrete generative models, marginalization, probabilistic models

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: A new type of discrete generative models that explicitly learn the marginals via marginalization self-consistency.

Abstract:

We introduce *marginalization models* (MAMs), a new family of generative models
for high-dimensional discrete data. They offer scalable and flexible generative modeling with tractable likelihoods
by explicitly modeling all induced marginal distributions. Marginalization models enable fast evaluation of arbitrary marginal probabilities with a single forward pass of the neural network, which overcomes a major limitation of methods with exact marginal inference, such as autoregressive models (ARMs). We propose scalable methods for learning the marginals, grounded in the concept of “*marginalization self-consistency*”. Unlike previous methods, MAMs support scalable training of any-order generative models for high-dimensional problems under the setting of *energy-based training*, where the goal is to match the learned distribution to a given desired probability (specified
by an unnormalized (log) probability function such as energy function or reward function). We demonstrate the effectiveness of the proposed model on a variety of discrete data distributions, including binary images, language, physical systems, and molecules, for *maximum likelihood* and *energy-based training* settings. MAMs achieve orders of magnitude speedup in evaluating the marginal probabilities on both settings. For energy-based training tasks, MAMs enable any-order generative modeling of high-dimensional problems beyond the capability of previous methods.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7731