Magnifying Three Phases of GAN Training via Evolution of Discriminator and Generator Gradients
Abstract: Generative Adversarial Networks (GANs) are powerful generative models but often suffer from mode mixture and mode collapse. We propose a three-phase characterization of GAN training: fitting, refining, and collapsing, where mode mixture and mode collapse are treated as inter-connected. Inspired by the particle model interpretation of GANs, we leverage the discriminator gradient to analyze particle movement and the generator gradient, specifically "steepness," to quantify the severity of mode mixture by measuring the generator's sensitivity to changes in the latent space. Using these theoretical insights into evolution of gradients, we design a specialized metric that integrates both gradients to detect the transition from refining to collapsing. This metric forms the basis of an early stopping algorithm, which stops training at a point that balances sample quality and diversity. Experiments on synthetic and real-world datasets, including MNIST, Fashion MNIST, and CIFAR-10, validate our theoretical findings and demonstrate the effectiveness of the proposed algorithm.
Submission Length: Regular submission (no more than 12 pages of main content)
Previous TMLR Submission Url: https://openreview.net/forum?id=MWam9EhQYS
Changes Since Last Submission: # Major Changes
- **Title**
The title has been updated from *"Magnifying the Three Phases of GAN Training — Fitting, Refining, and Collapsing"* to *"Magnifying Three Phases of GAN Training via Evolution of Discriminator and Generator Gradients"* to better highlight the theoretical tools introduced.
- **Abstract**
The abstract has been revised to emphasize the theoretical tools, particularly the roles of discriminator gradients and generator gradients.
- **Section 1: Introduction**
This section has been thoroughly rewritten to integrate the original motivation section, creating a more cohesive narrative. A new Table 1 has been added to summarize the proposed three phases of GAN training. The contributions have been clarified to provide a more precise description.
- **Section 2: Technical Preliminaries and Basic Assumptions**
A new section has been added to consolidate the theoretical foundations, including the roles of discriminator and generator gradients, as well as the basic assumptions. The particle model interpretation of GANs is explicitly credited to Yi et al. (2023). Furthermore, the definition of the generator gradient (i.e., steepness) has been updated from the global supremum of the max norm of the generator's Jacobian to the spectral norm at a local point, making it invariant under orthogonal coordinate transformations.
- **Section 4: The Second Phase of GAN Training — Refining**
Since the definition of steepness has been modified to the spectral norm at a local point, all theorems in this section have been updated to align with this new definition. In particular, Theorem 4.2 has been refined to account for cases where $\bar{x}$ matches one of the $x_i$'s, ensuring the lower bound of steepness remains nonconstant by introducing a new parameter $\lambda$.
- **Section 5: The Third Phase of GAN Training — Collapsing**
This section has been rewritten to provide deeper theoretical insights into the collapsing phase, with a focus on the interaction between discriminator and generator gradients (Theorem 5.1). The properties of generator gradients during the collapsing phase have been formally established (Theorem 5.2), forming the basis for improving the early stopping algorithm (Algorithm 2), which now jointly monitors both generator and discriminator gradients for enhanced stability.
- **Section 6: Experiments**
Figure 5 has been refined to include plots showing the evolution of steepness across epochs, providing an additional metric for detecting collapsing in the early stopping algorithm. Comparisons between steepness, FID scores, and duality gaps have been added in Figure 6 and the appendix.
- **Section 7: Conclusion**
The conclusion has been revised to better summarize the findings.
- **Appendix**
A roadmap has been added at the beginning of the appendix to provide a clear overview of its organization and structure.
---
# Minor Changes
- **Streamlined Figures**
Several conceptual illustrations have been removed to enhance clarity and maintain focus.
- **Reduced Training Runs**
Additional training runs have been omitted to improve conciseness and avoid an overly lengthy presentation.
- **Writing Improvements**
General improvements have been made to ensure clarity, readability, and flow throughout the manuscript.
- **Typographical Corrections**
Various typos and formatting inconsistencies have been fixed.
Assigned Action Editor: ~Fernando_Perez-Cruz1
Submission Number: 3757
Loading