Debiasing Evidence Approximations: On Importance-weighted Autoencoders and Jackknife Variational Inference

Sebastian Nowozin

Abstract:

The importance-weighted autoencoder (IWAE) approach of Burda et al. defines a sequence of increasingly tighter bounds on the marginal likelihood of latent variable models. Recently, Cremer et al. reinterpreted the IWAE bounds as ordinary variational evidence lower bounds (ELBO) applied to increasingly accurate variational distributions. In this work, we provide yet another perspective on the IWAE bounds. We interpret each IWAE bound as a biased estimator of the true marginal likelihood where for the bound defined on $K$ samples we show the bias to be of order O(1/K). In our theoretical analysis of the IWAE objective we derive asymptotic bias and variance expressions. Based on this analysis we develop jackknife variational inference (JVI),
a family of bias-reduced estimators reducing the bias to $O(K^{-(m+1)})$ for any given m < K while retaining computational efficiency. Finally, we demonstrate that JVI leads to improved evidence estimates in variational autoencoders. We also report first results on applying JVI to learning variational autoencoders.

Our implementation is available at https://github.com/Microsoft/jackknife-variational-inference

TL;DR: Variational inference is biased, let's debias it.

Keywords: variational inference, approximate inference, generative models

Code: [ Microsoft/jackknife-variational-inference](https://github.com/Microsoft/jackknife-variational-inference)