Semi-Implicit Variational Inference via Kernelized Path Gradient Descent

Tobias Pielok, Bernd Bischl, David Rügamer

TL;DR: We propose a kernelized KL estimator for semi-implicit variational inference that reduces bias and variance in high dimensions, overcomes mode blindness, and connects theoretically to amortized SVGD

Abstract: Semi-implicit variational inference (SIVI) is a powerful framework for approximating complex posterior distributions, but training with the Kullback–Leibler (KL) divergence can be challenging due to high variance and bias in high-dimensional settings. While current state-of-the-art score-based methods, particularly Kernel Semi-Implicit Variational Inference (K-SIVI), have been shown to also work in high dimensions, they can be "blind'' to isolated components and mixing proportions, especially in multi-modal distributions. In this work, we propose a kernelized KL divergence estimator that stabilizes training through nonparametric smoothing, effectively addressing the "blindness'' challenge. To further reduce the bias, we introduce an importance sampling correction. We provide a theoretical connection to the amortized version of the Stein variational gradient descent, which estimates the score gradient via Stein's identity, showing that both methods minimize the same objective, but our semi-implicit approach achieves lower gradient variance. In addition, our method's bias in function space is benign, leading to more stable and efficient optimization. Empirical results demonstrate that our method outperforms or matches state-of-the-art score matching methods in both performance and training efficiency.

Submission Number: 2