About¶

GMMVI (Gaussian Mixture Model Variational Inference) is a framework for optimizing a Gaussian mixture model $q(\mathbf{x})$ with Gaussian components $q(\mathbf{x}|o)$ and weights $q(o)$ ,

$q(\mathbf{x}) = \sum_o q(o) q(\mathbf{x}|o),$

with respect to the optimization problem,

$\max_{q(\mathbf{x})} \mathbb{E}_{q(\mathbf{x})} \left[ r(\mathbf{x}) \right] + \text{H}(q(\mathbf{x})),$

where $\text{H}(q(\mathbf{x}))$ denotes the mixture’s entropy and $r(\mathbf{x})$ assigns a reward to the sample $\mathbf{x}$ . If $r(\mathbf{x})$ corresponds to the energy of a Gibbs-Boltzmann distribution $p(\mathbf{x}) \propto \exp(r(\mathbf{x}))$ , the learned GMM will approximate the target distribution $p(\mathbf{x})$ by minimizing the reverse Kullback-Leibler divergence $\text{KL}\left(q(\mathbf{x})||p(\mathbf{x})\right)$ .

The optimization is performed with respect to the weights, means and covariance matrices, and if desired the number of components. The framework is build on Tensorflow 2, however, the reward function can also be implemented using different libraries, such as PyTorch.

Method¶

The optimization is performed iteratively, where at every iteration an independent natural gradient descent step is performed to the categorical distribution over weights, $q(o)$ , and to each individual component $q(\mathbf{x}|o)$ . This procedure was concurrently proposed by Arenz et al. [AZN18] and Lin et al. [LKS19a]. However, both approaches differ quite significantly in several design choices (e.g. how the natural gradients are estimated) and derived the procedure from different perspectives with different theoretical guarantees, and therefore the equivalence of both approaches was initially not understand. This framework is published along with the article that first established the close connection between both approaches, and was used to systematically evaluate the effects of the different design choices.

Design Choices¶

We distinguish design choices for seven different modules corresponding to the abstract classes, where for each design choice, there are two to three option implemented as concrete classes.

Naming Convention¶

Depending on which option is used for each design choice, there are currently 432 different instantiation supported by GMMVI. When referring to a specific instantiation, we use 7-letter codewords, where the presence of a letter implies, that the corresponding option was chosen. The mapping from letter to option is given in the following table:

Using this naming convention, ZAMTRUX refers to VIPS [AZN20], and SEPIFUX refers to the method by Lin et al. [LKS19a]. The recommended setting, however, is SAMTRON.

Module	Options
NgEstimator	MORE (Z)	Stein (S)
ComponentAdaptation	Fixed (E)	VIPS (A)
SampleSelector	Lin (P)	VIPS (M)
NgBasedComponentUpdater	Direct (I)	iBLR (Y)	Trust-Region (T)
ComponentStepsizeAdaptation	Fixed (F)	Decaying (D)	Adaptive (R)
WeightUpdater	Direct (U)	Trust-Region (O)
WeightStepsizeAdaptation	Fixed (X)	Decaying (G)	Adaptive (N)