Guiding Time-Varying Generative Models with Natural Gradients on Exponential Family Manifold

Song Liu, Leyang Wang, Yakun Wang

Track: long paper (up to 8 pages)

Keywords: time score matching, exponential family, natural gradient descent, generative modelling

TL;DR: We align the evolution of a generative model to natural gradient descent update on an exponential family manifold.

Abstract:

Optimising probabilistic models is a well-studied field in statistics. However, its connection with the training of generative models
remains largely under-explored. In this paper, we show that the evolution of time-varying generative models can be projected onto an exponential family manifold, naturally creating a link between the parameters of a generative model and those of a probabilistic model. We then train the generative model by moving its projection on the manifold according to the natural gradient descent scheme.
This approach also allows us to approximate the natural gradient of the KL divergence efficiently without relying on MCMC for intractable models. Furthermore, we propose particle versions of the algorithm, which feature closed-form update rules for any parametric model within the exponential family. Through toy and real-world experiments, we validate the effectiveness of the proposed algorithms. The code of the proposed method could be found at
\url{https://github.com/anewgithubname/iNGD}.

Submission Number: 101