On the Lipschitz Regularity of Optimal Discriminators

Karthik Srikumar, Aryaman Singh

Keywords: Generative Adversarial Networks, Mode Collapse, Lipschitz Regularity, Training Dynamics, Optimal Discriminator, Stability Analysis, Generative Models

TL;DR: We prove that the Lipschitz constant of the optimal discriminator in a GAN diverges to infinity in regions where the generator has negligible support relative to the true data distribution.

Abstract: The training dynamics of Generative Adversarial Networks (GANs) are, at their core, driven by the interplay between the generator and discriminator. One of the biggest challenges in analyzing the theory of GANs is the problem of mode collapse, in which the generator is unable to capture the full range of variability of the target distribution. This paper presents a formal mathematical analysis of the relationship between the Lipschitz constant of the optimal discriminator and the stability of gradient-based training and the problem of mode collapse. We show that, under non-parametric assumptions, the optimal discriminator for a given generator has a Lipschitz constant that grows unbounded in regions where the support of the generator distribution is negligible compared to the data distribution. This fundamental irregularity of the training dynamics causes the gradient updates of the generator to become unstable, giving a theoretical explanation for mode collapse that is agnostic to architectural and algorithmic details. We also present a sufficient condition on the support of the generator to guarantee that the optimal discriminator has a bounded Lipschitz constant, giving a new theoretical explanation for the use of gradient penalty and spectral normalization.

Submission Number: 2