Variational Lossy Autoencoder

Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, Pieter Abbeel

Abstract:

Representation learning seeks to expose certain aspects of observed data in a learned representation that's amenable to downstream tasks like classification.
For instance, a good representation for 2D images might be one that describes only global structure and discards information about detailed texture.
In this paper, we present a simple but principled method to learn such global representations by combining Variational Autoencoder (VAE) with neural autoregressive models such as RNN, MADE and PixelRNN/CNN.
Our proposed VAE model allows us to have control over what the global latent code can learn and , by designing the architecture accordingly, we can force the global latent code to discard irrelevant information such as texture in 2D images, and hence the code only ``autoencodes'' data in a lossy fashion.
In addition, by leveraging autoregressive models as both prior distribution $p(z)$ and decoding distribution $p(x|z)$, we can greatly improve generative modeling performance of VAEs, achieving new state-of-the-art results on MNIST, OMNIGLOT and Caltech-101 as well as competitive results on CIFAR10.

TL;DR: A VAE that provably learns global structure of images with a local PixelCNN decoder.

Conflicts: openai.com, berkeley.edu, mit.edu

Keywords: Deep learning, Unsupervised Learning