About¶
GMMVI (Gaussian Mixture Model Variational Inference) is a framework for optimizing a Gaussian mixture model
with Gaussian components
and weights
,
with respect to the optimization problem,
where denotes the mixture’s entropy and
assigns a reward to the
sample
. If
corresponds to the energy of a Gibbs-Boltzmann distribution
, the learned GMM will approximate the target distribution
by minimizing the reverse Kullback-Leibler divergence
.
The optimization is performed with respect to the weights, means and covariance matrices, and if desired the number of components. The framework is build on Tensorflow 2, however, the reward function can also be implemented using different libraries, such as PyTorch.
Method¶
The optimization is performed iteratively, where at every iteration an independent natural gradient descent step is
performed to the categorical distribution over weights, , and to each individual component
. This procedure was concurrently proposed by Arenz et al. [AZN18] and Lin et al. [LKS19a].
However, both approaches differ quite significantly in several design choices (e.g. how the natural gradients are
estimated) and derived the procedure from different perspectives with different theoretical guarantees, and therefore
the equivalence of both approaches was initially not understand. This framework is published along with the article
that first established the close connection between both approaches, and was used to systematically evaluate the
effects of the different design choices.
Design Choices¶
We distinguish design choices for seven different modules corresponding to the abstract classes, where for each design choice, there are two to three option implemented as concrete classes.
Naming Convention¶
Depending on which option is used for each design choice, there are currently 432 different instantiation supported by GMMVI. When referring to a specific instantiation, we use 7-letter codewords, where the presence of a letter implies, that the corresponding option was chosen. The mapping from letter to option is given in the following table:
Module |
Options |
||
Using this naming convention, ZAMTRUX refers to VIPS [AZN20], and SEPIFUX refers to the method by Lin et al. [LKS19a]. The recommended setting, however, is SAMTRON.