Posterior Collapse and Latent Variable Non-identifiability

Yixin Wang, David Blei, John Patrick Cunningham

Keywords: deep generative model, posterior collapse, latent variable identifiability

Abstract:

Variational autoencoders model high-dimensional data by positing
low-dimensional latent variables that are mapped through a flexible
distribution parametrized by a neural network. Unfortunately,
variational autoencoders often suffer from posterior collapse: the
posterior of the latent variables is equal to its prior, rendering the
variational autoencoder useless as a means to produce meaningful
representations. Existing approaches to posterior collapse often
attribute it to the use of neural networks or optimization issues due
to variational approximation. In this paper, we consider posterior
collapse as a problem of latent variable non-identifiability. We prove
that the posterior collapses if and only if the latent variables are
non-identifiable in the generative model. This fact implies that
posterior collapse is not a phenomenon specific to the use of flexible
distributions or approximate inference. Rather, it can occur in
classical probabilistic models even with exact inference, which we
also demonstrate. Based on these results, we propose a class of
latent-identifiable variational autoencoders, deep generative models
which enforce identifiability without sacrificing flexibility. This
model class resolves the problem of latent variable
non-identifiability by leveraging bijective Brenier maps and
parameterizing them with input convex neural networks, without special
variational inference objectives or optimization tricks. Across
synthetic and real datasets, latent-identifiable variational
autoencoders outperform existing methods in mitigating posterior
collapse and providing meaningful representations of the data.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

TL;DR: We show that posterior collapse is a problem of latent variable identifiability. We propose a class of VAE whose latent variables are guaranteed to be identifiable.

Code: https://github.com/yixinwang/lidvae-public