Variational Inference with Gaussian Mixture by Entropy Approximation
Abstract: Variational inference is a technique for approximating intractable posterior distributions in order to quantify the uncertainty of machine learning.
Although the unimodal Gaussian distribution is usually chosen as a variational family, it hardly approximates the multimodality. In this paper, we employ the Gaussian mixture distribution as a variational family. A main difficulty of variational inference with the Gaussian mixture is how to approximate the entropy of the Gaussian mixture. We approximate the entropy of the Gaussian mixture as the sum of the entropy of the unimodal Gaussian, which can be analytically calculated. In addition, we theoretically analyze the approximation error between the true entropy and approximated one in order to reveal when our approximation works well. Specifically, the approximation error is controlled by the ratios of the distances between the means to the sum of the variances of the Gaussian mixture. Furthermore, it converges to zero when the ratios go to infinity. This situation seems to be more likely to occur in higher dimensional parametric spaces because of the curse of dimensionality. Therefore, our result guarantees that our approximation works well, for example, in neural networks that assume a large number of weights.
Submission Length: Regular submission (no more than 12 pages of main content)
Assigned Action Editor: ~Pierre_Alquier1
Submission Number: 1983
Loading