Perturb and Learn: Energy-Based Modelling in Discrete Spaces without MCMC

Tobias Schröder, Zijing Ou, Yingzhen Li, Andrew B. Duncan

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Energy-based models, discrete probabilistic modelling, importance sampling

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We propose training discrete Energy-Based Models (EBMs) with Energy Discrepancy, which enables EBM training with tractable importance sampling instead of MCMC.

Abstract: Energy-based models (EBMs) offer a flexible framework for probabilistic modelling across various data domains. However, training EBMs on discrete data poses significant challenges, primarily due to the intricacies of sampling in such spaces. In this work, we propose to train discrete EBMs with Energy Discrepancy which only requires the evaluation of the energy function at data points and their perturbed counterparts, thus eliminating the need for demanding sampling techniques like Markov chain Monte Carlo. Energy discrepancy offers theoretical guarantees applicable to a broad class of perturbation processes, of which we investigate three types: perturbations based on Bernoulli noise, deterministic transforms, and neighbourhood structures. We estimate the energy discrepancy loss effectively using importance sampling with two types of proposal distributions: uninformed and gradient-informed. Empirically, we demonstrate the efficacy of the proposed approaches in a wide range of applications, including Ising models training, discrete density estimation, graph generation, and discrete image modelling.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7434