Degeneration in VAE: in the Light of Fisher Information Loss

Huangjie Zheng, Jiangchao Yao, Ya Zhang, Ivor W. Tsang

Feb 12, 2018 (modified: Feb 12, 2018) ICLR 2018 Workshop Submission readers: everyone
  • Abstract: Variational Autoencoder (VAE) is one of the most popular generative models, and enormous advances have been explored in recent years. Due to the increasing complexity of raw data and model architecture, deep networks need to deploy in VAE while few works discuss their impacts. According to our observation, VAEs do not always benefit from deeper architecture: 1) Deeper encoder makes VAE learn more comprehensible latent representation, while results in blurry samples; 2) Deeper decoder ensures more high-quality generation, but latent representation becomes abstruse; 3) When encoder and decoder both go deeper, abstruse latent representation occurs with blurry samples at same time. In this paper, we deduce a Fisher information measure for the corresponding analysis. With such measure, we demonstrate that the information loss is ineluctable in feed-forward network and causes the previous three types of degeneration, especially when the networks go deeper. We also demonstrate that skip connections benefit information preservation, thus propose a VAE enhanced by skip connections, named SCVAE. In the experiments, SCVAE is shown to mitigate information loss and achieve a promising performance in both encoding and decoding tasks.
  • TL;DR: We discuss the impacts of deeper encoder and decoder in VAE, and propose a solution for the problems caused when VAE goes deeper.
  • Keywords: Variational AutoEncoder, Neural Networks, Skip Connection