Bridging VMP and CEP: Theoretical Insights for Connecting Different Approximate Bayesian Inference Methods

TMLR Paper6740 Authors

Abstract: Approximate Bayesian inference (ABI) methods have become indispensable tools in modern machine learning and statistics for approximating intractable posterior distributions. Despite the related extensive studies and applications across diverse domains, the theoretical connections among these methods have remained relatively unexplored. This paper takes the first step to uncover the underlying relationships between two widely employed ABI techniques: the variational message passing (VMP) and the conditional expectation propagation (CEP) methods. Through rigorous mathematical analysis, we demonstrate a strong connection between these two approaches under mild conditions, from optimization as well as graphical model perspectives. This newly unveiled connection not only enhances our understanding of the performance and convergence properties of VMP and CEP, but it also facilitates the cross-fertilization of their respective strengths. For instance, we establish the convergence of CEP under mild conditions and demonstrate how this connection facilitates the construction of streaming VMP. Furthermore, our findings provide insights into the underlying relationships and distinctive characteristics of other ABI methods, shedding new light on the understanding and development of more advanced ABI techniques. To validate our theoretical findings, we derive and analyze various ABI methods within the context of Bayesian tensor decomposition, a fundamental tool in machine learning research. Specifically, we show that these two approaches yield the same updates within this context and illustrate how the established connection can be leveraged to construct a streaming version of the VMP-based Bayesian tensor decomposition algorithm.

Submission Type: Long submission (more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=alODfvLNuP&referrer=%5BAuthor%20Console%5D(%2Fgroup%3Fid%3DTMLR%2FAuthors%23your-submissions)

Changes Since Last Submission:

In the previous submission of this manuscript, we received professional and constructive comments from three reviewers. We sincerely thank them for their feedback and the Action Editor for the opportunity to resubmit. Guided by their insightful comments and the Action Editor’s decision, we have performed major revisions to improve the clarity, rigor, and completeness of our manuscript. The key changes are summarized as follows:

- **Refinement of Theoretical Rigor and Narrative Consistency**: Incorporating the insightful suggestions regarding theoretical accuracy, we have meticulously revised Theorem 1 and Lemma 4 to rigorously characterize the equivalence using the delta approximation and sufficient statistics reparameterization. We corrected the notation for natural parameters to avoid potential Jensen gap issues and ensure mathematical precision. Furthermore, we ensured that the manuscript's narrative fully aligns with these theoretical foundations. We also recalibrated our claims throughout the text by explicitly framing the contribution as an algorithmic equivalence between VMP and CEP under specific conditions, rather than a broad unification of VI and EP. This ensures the theoretical presentation is accurate, complete, and self-consistent.

- **Enhancement of Structure and Empirical Validation**: To further improve clarity, we restructured the background sections. This includes adding a new subsection comparing VI- and EP-based methods and moving foundational lemmas to the Appendix. We also added illustrative figures, such as Bayesian networks and factor graphs, to support conceptual understanding. Empirically, we added Section 4.5 to present numerical results for Bayesian tensor decomposition. Moreover, we included new robustness experiments, specifically testing entry-wise updates and non-i.i.d. noise, to demonstrate that the algorithms perform reliably even when theoretical assumptions are relaxed.

- **Discussion on Divergence Minimization**: Following the Action Editor's suggestion, we incorporated a detailed discussion on the divergence minimization perspective, such as Power EP and Rényi Divergence VI, in Sections 1 and 2.4. We explicitly distinguish our contribution from these existing frameworks: while they unify methods at a general objective level, our work focus on their variants and establishes an intrinsic algorithmic-level equivalence under specific conditions.

We believe these revisions comprehensively address the concerns raised and significantly strengthen the quality of the manuscript.

Assigned Action Editor: ~Yingzhen_Li1

Submission Number: 6740