Asymptotic Free Energy Of Variational Bayesian Deep Learning

Hiroshi Wakimori, Tikara Hosino

Abstract: The Bayesian deep learning is promising for its theoretical foundation. Especially, it was probed that free energy can asymptotically identify the structure of the true distribution consistently. In this paper, we derive the asymptotic expected variational free energy in the case of Gaussian trial posterior. The result shows that the variance of the posterior reflects the relative structure of the true distribution and the learning model. This result clarifies the theoretical insights of model selection and model distillation in variational approximation of Bayesian methods.

Keywords: variational inference, free energy, deep learning, model selection, model distillation

TL;DR: We derive the asymptotic expected variational free energy of the Bayesian deep learning in the case of Gaussian trial posterior.