EBM Life Cycle: MCMC Strategies for Synthesis, Defense, and Density Modeling

Mitch Hill, Jonathan Craig Mitchell, Chu Chen, Yuan Du, Mubarak Shah, Song-Chun Zhu

Keywords: energy-based model, MCMC sampling, Langevin sampling, generative modeling, unsupervised learning, image synthesis, adversarial defense, density estimation

Abstract: This work presents strategies to learn an Energy-Based Model (EBM) according to the desired length of its MCMC sampling trajectories. MCMC trajectories of different lengths correspond to models with different purposes. Our experiments cover three different trajectory magnitudes and learning outcomes: 1) shortrun sampling for image generation; 2) midrun sampling for classifier-agnostic adversarial defense; and 3) longrun sampling for principled modeling of image probability densities. To achieve these outcomes, we introduce three novel methods of MCMC initialization for negative samples used in Maximum Likelihood (ML) learning. With standard network architectures and an unaltered ML objective, our MCMC initialization methods alone enable significant performance gains across the three applications that we investigate. Our results include state-of-the-art FID scores for unnormalized image densities on the CIFAR-10 and ImageNet datasets; state-of-the-art adversarial defense on CIFAR-10 among purification methods and the first EBM defense on ImageNet; and scalable techniques for learning valid probability densities.

One-sentence Summary: Our works explores new MCMC initialization methods for the EBM learning process to obtain shortrun models for image synthesis, midrun models for adversarial defense, and longrun models for density modeling.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2205.12243/code)