Provable Convergence and Global Optimality of Generative Adversarial Network

Qi Cai, Zhuoran Yang, Jason D. Lee, Shaolei S. Du, Zhaoran Wang

Keywords: generative adversarial network, IPM-based GANs, overparametrized neural network

TL;DR: We establish global convergence to optimality for IPM-based GANs where the generator is an overparametrized neural network.

Abstract: Generative adversarial networks (GANs) train implicit generative models through solving minimax problems. Such minimax problems are known as nonconvex- nonconcave, for which the dynamics of first-order methods are not well understood. In this paper, we consider GANs in the type of the integral probability metrics (IPMs) with the generator represented by an overparametrized neural network. When the discriminator is solved to approximate optimality in each iteration, we prove that stochastic gradient descent on a regularized IPM objective converges globally to a stationary point with a sublinear rate. Moreover, we prove that when the width of the generator network is sufficiently large and the discriminator function class has enough discriminative ability, the obtained stationary point corresponds to a generator that yields a distribution that is close to the distribution of the observed data in terms of the total variation. To the best of our knowledge, we seem to first establish both the global convergence and global optimality of training GANs when the generator is parametrized by a neural network.