About

GMMVI (Gaussian Mixture Model Variational Inference) is a framework for optimizing a Gaussian mixture model q(\mathbf{x}) with Gaussian components q(\mathbf{x}|o) and weights q(o),

q(\mathbf{x}) = \sum_o q(o) q(\mathbf{x}|o),

with respect to the optimization problem,

\max_{q(\mathbf{x})} \mathbb{E}_{q(\mathbf{x})} \left[ r(\mathbf{x}) \right] + \text{H}(q(\mathbf{x})),

where \text{H}(q(\mathbf{x})) denotes the mixture’s entropy and r(\mathbf{x}) assigns a reward to the sample \mathbf{x}. If r(\mathbf{x}) corresponds to the energy of a Gibbs-Boltzmann distribution p(\mathbf{x}) \propto \exp(r(\mathbf{x})), the learned GMM will approximate the target distribution p(\mathbf{x}) by minimizing the reverse Kullback-Leibler divergence \text{KL}\left(q(\mathbf{x})||p(\mathbf{x})\right).

The optimization is performed with respect to the weights, means and covariance matrices, and if desired the number of components. The framework is build on Tensorflow 2, however, the reward function can also be implemented using different libraries, such as PyTorch.

Method

The optimization is performed iteratively, where at every iteration an independent natural gradient descent step is performed to the categorical distribution over weights, q(o), and to each individual component q(\mathbf{x}|o). This procedure was concurrently proposed by Arenz et al. [AZN18] and Lin et al. [LKS19a]. However, both approaches differ quite significantly in several design choices (e.g. how the natural gradients are estimated) and derived the procedure from different perspectives with different theoretical guarantees, and therefore the equivalence of both approaches was initially not understand. This framework is published along with the article that first established the close connection between both approaches, and was used to systematically evaluate the effects of the different design choices.

Design Choices

We distinguish design choices for seven different modules corresponding to the abstract classes, where for each design choice, there are two to three option implemented as concrete classes.

  1. NgEstimator
    Inheritance diagram of gmmvi.optimization.gmmvi_modules.ng_estimator.NgEstimator, gmmvi.optimization.gmmvi_modules.ng_estimator.SteinNgEstimator, gmmvi.optimization.gmmvi_modules.ng_estimator.MoreNgEstimator
  2. ComponentAdaptation
    Inheritance diagram of gmmvi.optimization.gmmvi_modules.component_adaptation.ComponentAdaptation, gmmvi.optimization.gmmvi_modules.component_adaptation.FixedComponentAdaptation, gmmvi.optimization.gmmvi_modules.component_adaptation.VipsComponentAdaptation
  3. SampleSelector
    Inheritance diagram of gmmvi.optimization.gmmvi_modules.sample_selector.SampleSelector, gmmvi.optimization.gmmvi_modules.sample_selector.VipsSampleSelector, gmmvi.optimization.gmmvi_modules.sample_selector.LinSampleSelector
  4. NgBasedComponentUpdater
    Inheritance diagram of gmmvi.optimization.gmmvi_modules.ng_based_component_updater.NgBasedComponentUpdater, gmmvi.optimization.gmmvi_modules.ng_based_component_updater.DirectNgBasedComponentUpdater, gmmvi.optimization.gmmvi_modules.ng_based_component_updater.NgBasedComponentUpdaterIblr, gmmvi.optimization.gmmvi_modules.ng_based_component_updater.KLConstrainedNgBasedComponentUpdater
  5. ComponentStepsizeAdaptation
    Inheritance diagram of gmmvi.optimization.gmmvi_modules.component_stepsize_adaptation.ComponentStepsizeAdaptation, gmmvi.optimization.gmmvi_modules.component_stepsize_adaptation.FixedComponentStepsizeAdaptation, gmmvi.optimization.gmmvi_modules.component_stepsize_adaptation.DecayingComponentStepsizeAdaptation, gmmvi.optimization.gmmvi_modules.component_stepsize_adaptation.ImprovementBasedComponentStepsizeAdaptation
  6. WeightUpdater
    Inheritance diagram of gmmvi.optimization.gmmvi_modules.weight_updater.WeightUpdater, gmmvi.optimization.gmmvi_modules.weight_updater.TrustRegionBasedWeightUpdater, gmmvi.optimization.gmmvi_modules.weight_updater.DirectWeightUpdater
  7. WeightStepsizeAdaptation
    Inheritance diagram of gmmvi.optimization.gmmvi_modules.weight_stepsize_adaptation.WeightStepsizeAdaptation, gmmvi.optimization.gmmvi_modules.weight_stepsize_adaptation.FixedWeightStepsizeAdaptation, gmmvi.optimization.gmmvi_modules.weight_stepsize_adaptation.DecayingWeightStepsizeAdaptation, gmmvi.optimization.gmmvi_modules.weight_stepsize_adaptation.ImprovementBasedWeightStepsizeAdaptation

Naming Convention

Depending on which option is used for each design choice, there are currently 432 different instantiation supported by GMMVI. When referring to a specific instantiation, we use 7-letter codewords, where the presence of a letter implies, that the corresponding option was chosen. The mapping from letter to option is given in the following table:

Module

Options

NgEstimator

MORE (Z)

Stein (S)

ComponentAdaptation

Fixed (E)

VIPS (A)

SampleSelector

Lin (P)

VIPS (M)

NgBasedComponentUpdater

Direct (I)

iBLR (Y)

Trust-Region (T)

ComponentStepsizeAdaptation

Fixed (F)

Decaying (D)

Adaptive (R)

WeightUpdater

Direct (U)

Trust-Region (O)

WeightStepsizeAdaptation

Fixed (X)

Decaying (G)

Adaptive (N)

Using this naming convention, ZAMTRUX refers to VIPS [AZN20], and SEPIFUX refers to the method by Lin et al. [LKS19a]. The recommended setting, however, is SAMTRON.