Revisiting Unbiased Implicit Variational Inference

Tobias Pielok, Bernd Bischl, David Rügamer

TL;DR: We propose a competitive new SIVI method by identifying key shortcomings of the Unbiased Implicit Variational Inference method.

Abstract:

Recent years have witnessed growing interest in semi-implicit variational inference (SIVI) methods due to their ability to rapidly generate samples from complex distributions. However, since the likelihood of these samples is non-trivial to estimate in high dimensions, current research focuses on finding effective SIVI training routines.
Although unbiased implicit variational inference (UIVI) has largely been dismissed as imprecise and computationally prohibitive because of its inner MCMC loop, we revisit this method
and show that UIVI's MCMC loop can be effectively replaced via importance sampling and the optimal proposal distribution can be learned stably by minimizing an expected forward Kullback–Leibler divergence without bias. Our refined approach demonstrates superior performance or parity with state-of-the-art methods on established SIVI benchmarks.

Lay Summary:

Modern AI systems often need to estimate uncertainty — especially when working with limited or noisy data. A common way to do this is through variational inference, which approximates complex probability distributions using simpler ones. A powerful version called semi-implicit variational inference (SIVI) can represent rich and flexible distributions but is difficult to train efficiently.

An earlier approach, unbiased implicit variational inference (UIVI), offered strong theoretical guarantees but was largely abandoned due to its reliance on slow simulation methods. In this work, we revisit UIVI and replace its slow component with a more scalable technique: instead of running simulations, we draw samples from a carefully chosen distribution and give each sample a weight based on how relevant it is. This allows us to approximate the same results as the original method, but far more efficiently.

Our method matches or improves upon state-of-the-art techniques on benchmark problems, making uncertainty estimation in machine learning more practical and reliable.

Primary Area: Probabilistic Methods->Variational Inference

Keywords: semi-implicit variational inference; path gradient; importance sampling; conditional normalizing flows

Submission Number: 5256