Conditional Noise-Contrastive Estimation of Energy-Based Models by Jumping Between Modes

Hanlin Yu, Michael U. Gutmann, Arto Klami, Omar Chehab

Keywords: Conditional Noise-Contrastive Estimation, Energy-Based Models, Diffusion Models, multi-modal

Abstract: Learning Energy-Based Models (EBMs) is notoriously difficult when the data distribution is multi-modal. Standard methods such as Score Matching --- even when amortized across many noisy versions of the data as in Energy-Based Diffusion Models --- often fail to capture relative energies between modes because they rely solely on local energy differences. We address this limitation by also considering global energy differences. To do so, we use Conditional Noise-Contrastive Estimation (CNCE) which estimates energy differences between pairs of points drawn using a freely chosen noise distribution. We design this noise distribution to propose pairs of points from different modes, thus comparing the modes directly. We further obtain the asymptotic estimation error of CNCE, derive a theoretically optimal noise distribution, and provide a practical algorithm that combines local and global energy differences. Experiments show that this approach substantially improves estimation in multi-modal settings.

Submission Number: 12