Variational Inference with Singularity-Free Planar Flows

Jiangrong Ouyang, Mingming Gong, Howard Bondell

Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: variational inference, normalizing flow, planar flow, variational autoencoder

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: Variational inference is a method for approximating probability distributions. The approximation quality depends on the expressiveness of variational distributions. Normalizing flows provide a way to construct a flexible and rich family of distributions. Planar flow, an early studied normalizing flow, is simple but powerful. Our research reveals a crucial insight into planar flow's constrained parameters: they exhibit a non-removable singularity in their original reparameterization. The gradients of the associated parameters diverge to infinity in different directions as they approach to the singularity, which creates a potential for the model to overshoot and get stuck in some undesirable states. We then propose a new reparameterization to circumvent the singularity. The resulting singularity-free planar flows are more stable in training and demonstrate better performance in variational inference tasks.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5682