Hybrid Kernel Stein Variational Gradient Descent
Keywords: Stein Variational Gradient Descent, Approximate Inference, Particle-based Variational Inference, Gradient Flow
Abstract: Stein variational gradient descent (SVGD) is a particle based approximate inference algorithm. Many variants of SVGD have been proposed in recent years, including the hybrid kernel variant (h-SVGD), which has demonstrated promising results on image classification with deep neural network ensembles. In this paper, we demonstrate the ability of h-SVGD to alleviate variance collapse, a problem that SVGD is known to suffer from. Unlike other SVGD variants that alleviate variance collapse, h-SVGD does not incur additional computational cost, nor does it require the target density to factorise. We also develop the theory of h-SVGD by demonstrating the existence of a solution to the hybrid Stein partial differential equation. We highlight a special case in which h-SVGD is a kernelised Wasserstein gradient flow on a functional other than the Kullback-Leibler divergence, which is the functional describing the SVGD gradient flow. By characterising the fixed point in this special case, we show that h-SVGD does not converge to the target distribution in the the mean field limit. Other theoretical results include a descent lemma and a large particle limit result. Despite the bias in the mean field limiting distribution, experiments demonstrate that h-SVGD remains competitive on high dimensional inference tasks whilst alleviating variance collapse.
Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 5389
Loading