Natural Gradient Variational BayesWithout FisherMatrix Analytic Calculation and Its Inversion

Minh-Ngoc Tran

Abstract:

This article introduces a method for efficiently approximating the inverse of the Fisher information matrix,
a crucial step in achieving effective variational Bayes inference. A notable aspect of our approach is the
avoidance of analytically computing the Fisher information matrix and its explicit inversion. Instead, we
introduce an iterative procedure for generating a sequence of matrices that converge to the inverse of
Fisher information. The natural gradient variational Bayes algorithm without analytic expression of the
Fisher matrix and its inversion is provably convergent and achieves a convergence rate of order O(log s/s),
with s the number of iterations. We also obtain a central limit theorem for the iterates. Implementation of
our method does not require storage of large matrices, and achieves a linear complexity in the number
of variational parameters. Our algorithm exhibits versatility, making it applicable across a diverse array of
variational Bayes domains, including Gaussian approximation and normalizing flow Variational Bayes. We
offer a range of numerical examples to demonstrate the efficiency and reliability of the proposed variational
Bayes method. Supplementary materials for this article are available online, including a standardized
description of the materials available for reproducing the work.