Gaussian-Bernoulli RBMs Without Tears

Renjie Liao, Simon Kornblith, Mengye Ren, David J. Fleet, Geoffrey Hinton

Keywords: Gaussian-Bernoulli Restricted Boltzmann Machines, Restricted Boltzmann Machines, Langevin Monte Carlo Sampling, Contrastive Divergence

Abstract:

We revisit the challenging problem of training Gaussian-Bernoulli restricted Boltzmann machines (GRBMs), introducing two innovations. We propose a novel Gibbs-Langevin sampling algorithm that outperforms existing methods like Gibbs sampling.
We propose a modified contrastive divergence (CD) algorithm so that one can generate images with GRBMs starting from noise.
This enables direct comparison of GRBMs with deep generative models, improving evaluation protocols in the RBM literature.
Moreover, we show that modified CD and gradient clipping are enough to robustly train GRBMs with large learning rates, thus removing the necessity of various tricks in the literature.
Experiments on Gaussian Mixtures, MNIST, FashionMNIST, and CelebA show GRBMs can generate good samples, despite their single-hidden-layer architecture.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Generative models

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2210.10318/code)