Improved Contrastive Divergence Training of Energy Based Models

Yilun Du, Shuang Li, Joshua B. Tenenbaum, Igor Mordatch

Keywords: Contrastive Divergence, Energy Based Modeling

Abstract: We propose several different techniques to improve contrastive divergence training of energy-based models (EBMs). We first show that a gradient term neglected in the popular contrastive divergence formulation is both tractable to estimate and is important to avoid training instabilities in previous models. We further highlight how data augmentation, multi-scale processing, and reservoir sampling can be used to improve model robustness and generation quality. Thirdly, we empirically evaluate stability of model architectures and show improved performance on a host of benchmarks and use cases, such as image generation, OOD detection, and compositional generation.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: Improvements to contrastive divergence to allow better training of EBMs

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2012.01316/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=fTedmd6VLa