Variational Bayes on manifolds

Minh-Ngoc Tran

Abstract:

Variational Bayes (VB) has become a widely-used tool for Bayesian inference in statistics and machine learning. Nonetheless,
the development of the existing VB algorithms is so far generally restricted to the case where the variational parameter space
is Euclidean, which hinders the potential broad application of VB methods. This paper extends the scope of VB to the case
where the variational parameter space is a Riemannian manifold. We develop an efficient manifold-based VB algorithm
that exploits both the geometric structure of the constraint parameter space and the information geometry of the manifold
of VB approximating probability distributions. Our algorithm is provably convergent and achieves a convergence rate of
order O(1/
√
T ) and O(1/T 2−2) for a non-convex evidence lower bound function and a strongly retraction-convex evidence
lower bound function, respectively. We develop in particular two manifold VB algorithms, Manifold Gaussian VB and
ManifoldWishart VB, and demonstrate through numerical experiments that the proposed algorithms are stable, less sensitive
to initialization and compares favourably to existing VB methods.