Learning Equivariant Energy Based Models with Equivariant Stein Variational Gradient DescentDownload PDF

21 May 2021, 20:44 (edited 26 Oct 2021)NeurIPS 2021 PosterReaders: Everyone
  • Keywords: Energy Based Models, Equivariance, Equivariant Sampling, Stein Variational Gradient Descent, Unsupervised Learning, Self-Supervised Learning
  • TL;DR: We propose equivariant SVGD for sampling from invariant densities and use it to train equivariant energy based models
  • Abstract: We focus on the problem of efficient sampling and learning of probability densities by incorporating symmetries in probabilistic models. We first introduce Equivariant Stein Variational Gradient Descent algorithm -- an equivariant sampling method based on Stein's identity for sampling from densities with symmetries. Equivariant SVGD explicitly incorporates symmetry information in a density through equivariant kernels which makes the resultant sampler efficient both in terms of sample complexity and the quality of generated samples. Subsequently, we define equivariant energy based models to model invariant densities that are learned using contrastive divergence. By utilizing our equivariant SVGD for training equivariant EBMs, we propose new ways of improving and scaling up training of energy based models. We apply these equivariant energy models for modelling joint densities in regression and classification tasks for image datasets, many-body particle systems and molecular structure generation.
  • Supplementary Material: pdf
  • Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.
  • Code: https://github.com/priyankjaini/equivariant_contrastive_learning
13 Replies