Posterior Collapse and Latent Variable Non-identifiability

Yixin Wang, John Patrick Cunningham

Abstract:

Variational autoencoders (VAE) powerfully model high-dimensional data
by combining classical probabilistic modeling and neural networks. In
a VAE, each datapoint is generated by a low-dimensional latent
variable popped through some flexible implicit distribution
parametrized by neural networks. A VAE then employs variational
inference to infer the posterior of these latents. Though flexible,
VAEs often suffer from posterior collapse, a phenomenon where the
inferred posterior of the latent variables is equal to its
(uninformative) prior. In this paper, we study the posterior collapse
phenomenon from the lens of latent variable identifiability. We find
that posterior collapse can occur in classical probabilistic models
(e.g. Gaussian mixture models) fitted with Markov chain Monte Carlo
methods whenever their latent variables are not identifiable. It is
contrary to the belief that posterior collapse is specific to VAEs
that use neural networks or variational approximation. Mathematically,
we prove that posterior collapses if and only if the generative model
is non-identifiable. Further, a VAE is non-identifiable, hence
suffering from posterior collapse. Finally, we propose a class of
identifiable VAEs that leverage optimal transport maps to resolve the
latent variable non-identifiability. Across datasets, we show that the
identifiable VAEs prevents posterior collapse.