Implicit competitive regularization in GANs

Florian Schaefer, Hongkai Zheng, Anima Anandkumar

TL;DR: Training GANs by modeling networks as agents acting with limited information and in awareness of their opponent imposes an implicit regularization that leads to stable convergence and prevents mode collapse.

Abstract:

Generative adversarial networks (GANs) are capable of producing high quality samples, but they suffer from numerous issues such as instability and mode collapse during training. To combat this, we propose to model the generator and discriminator as agents acting under local information, uncertainty, and awareness of their opponent. By doing so we achieve stable convergence, even when the underlying game has no Nash equilibria. We call this mechanism \emph{implicit competitive regularization} (ICR) and show that it is present in the recently proposed \emph{competitive gradient descent} (CGD).
When comparing CGD to Adam using a variety of loss functions and regularizers on CIFAR10, CGD shows a much more consistent performance, which we attribute to ICR.
In our experiments, we achieve the highest inception score when using the WGAN loss (without gradient penalty or weight clipping) together with CGD. This can be interpreted as minimizing a form of integral probability metric based on ICR.

Keywords: GAN, competitive optimization, game theory

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:1910.05852/code)