Asymptotic Behavior of the Coordinate Ascent Variational Inference in Singular Models

Sean C Plummer, Anirban Bhattacharya, Debdeep Pati, Yun Yang

Keywords: Coordinate Ascent Variational Inference, Singular Models, Dynmaical Systems

TL;DR: We provide a case study the convergence behavior for block coordinate ascent variational inference for the representative form of a general singular model which is a highly nonconvex distribution.

Abstract: Mean-field approximations are widely used for efficiently approximating high-dimensional integrals. While the efficacy of such approximations is well understood for well-behaved likelihoods, it is not clear how accurately it can approximate the marginal likelihood associated with a highly non log-concave singular model. In this article, we provide a case study of the convergence behavior of coordinate ascent variational inference (CAVI) in the context of a general $d$-dimensional singular model in standard form. We prove that for a general $d$-dimensional singular model in standard form with real log canonical threshold (RLCT) $\lambda$ and multiplicity $m$, the CAVI system converges to one of $m$ locally attracting fixed points. Furthermore, at each of these fixed points, the evidence lower bound (ELBO) of the system recovers the leading-order behavior of the asymptotic expansion of the log marginal likelihood predicted by \citet{watanabe1999algebraic, watanabe2001algebraic, watanabe2001balgebraic}. Our empirical results demonstrate that for models with multiplicity $m=1$ the ELBO provides a tighter approximation to the log-marginal likelihood than the asymptotic approximation $-\lambda \log n + o( \log \log n)$ of \citet{watanabe1999algebraic}.

Submission Number: 69