Slashed Normal: Parameterize Normal Posterior Distributions with KL Amplitude

Yujia Yan, Xingjian Du, Zhiyao Duan

Keywords: Variational Inference, Kullback-Leibler Divergence, Posterior Parameterization, Variational Autoencoders, Variational Information Bottleneck

TL;DR: We propose a simple alternative parameterization to Gaussian latents that has L2 Norm of raw neural network outputs as its KL divergence value.

Abstract:

We present Slashed Normal, a novel parameterization for the normal posterior
distribution in variational-inference-based latent variable models. Slashed Normal
takes a simple form resembling conventional practice, but uses the new stdplus
activation function to derive the standard deviation instead of softplus or exp. Although taking this simple form, the Slashed Normal establishes a direct connection between the squared l2-norm of the raw neural network output, termed KL amplitude, and the exact KL divergence value between the prior and the posterior. As a result, this parameterization enables a direct control of the KL divergence value, which is usually interpreted as the rate from the rate-distortion perspective for variational
autoencoders. We demonstrate the versatility of Slashed Normal through theoretical analysis and experiments, showcasing its ability to provide good insight about the posterior distribution, explicit control over the KL divergence, and mitigate
posterior collapse.

Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 14208