A Variational Inequality Perspective on Generative Adversarial Networks

Gauthier Gidel, Hugo Berard, Gaëtan Vignoud, Pascal Vincent, Simon Lacoste-Julien

Abstract: Generative adversarial networks (GANs) form a generative modeling approach known for producing appealing samples, but they are notably difficult to train. One common way to tackle this issue has been to propose new formulations of the GAN objective. Yet, surprisingly few studies have looked at optimization methods designed for this adversarial training. In this work, we cast GAN optimization problems in the general variational inequality framework. Tapping into the mathematical programming literature, we counter some common misconceptions about the difficulties of saddle point optimization and propose to extend methods designed for variational inequalities to the training of GANs. We apply averaging, extrapolation and a computationally cheaper variant that we call extrapolation from the past to the stochastic gradient method (SGD) and Adam.

Keywords: optimization, variational inequality, games, saddle point, extrapolation, averaging, extragradient, generative modeling, generative adversarial network

TL;DR: We cast GANs in the variational inequality framework and import techniques from this literature to optimize GANs better; we give algorithmic extensions and empirically test their performance for training GANs.

Code: [ GauthierGidel/Variational-Inequality-GAN](https://github.com/GauthierGidel/Variational-Inequality-GAN)

Data: [CIFAR-10](https://paperswithcode.com/dataset/cifar-10)

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:1802.10551/code)