Entropy-Guided Sampling of Flat Modes in Discrete Spaces

Abstract:

Sampling from flat modes in discrete spaces is a crucial yet under-explored direction. Flat modes represent robust solutions and have broad applications in combinatorial optimization and discrete generative modeling. However, existing sampling algorithms often overlook the mode volume and struggle to capture flat modes effectively. To address this limitation, we propose Entropic Discrete Langevin Proposal (EDLP), which incorporates local entropy into the sampling process through a continuous auxiliary variable under a joint distribution. The
local entropy term guides the discrete sampler toward flat modes with a small overhead. We provide non-asymptotic convergence guarantees for EDLP in locally log-concave discrete distributions. Empirically, our method consistently outperforms traditional approaches across tasks that require sampling from flat basins, including Bernoulli distributions, restricted Boltzmann machines, combinatorial optimization, and binary neural networks.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Chris_J_Maddison1

Submission Number: 7742