Bridging the Gap Between f-divergences and Bayes Hilbert Spaces

Linus Lach, Alexander Willi Fottner, Yarema Okhrin

Keywords: f-divergences, Bayes Hilbert spaces, fenchel conjugates, centered-log-ratio, posterior approximation

TL;DR: We generalize f-divergences to bridge the gap to Bayes Hilbert spaces by introducing a novel pseudo divergence framework that allows locally non-convex divergence generating functions.

Abstract:

We introduce a novel framework that generalizes $f$-divergences by incorporating locally non-convex divergence-generating functions.
Using this extension, we define a new class of pseudo $f$-divergences, encompassing a wider range of distances between distributions that traditional $f$-divergences cannot capture.
Among these, we focus on a particular pseudo divergence obtained by considering the induced metric of Bayes Hilbert spaces.
Bayes Hilbert spaces are frequently used due to their inherent connection to Bayes's theorem. They allow sampling from potentially intractable posterior densities, which has remained challenging until now.
In the more general context, we prove that pseudo $f$-divergences are well-defined and introduce a variational estimation framework that can be used in a statistical learning context.
By applying this variational estimation framework to $f$-GANs, we achieve improved FID scores over existing $f$-GAN architectures and competitive results with the Wasserstein GAN, highlighting its potential for both theoretical research and practical applications in learning theory.

Primary Area: learning theory

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8099