Sample weighting as an explanation for mode collapse in generative adversarial networks

Aksel Wilhelm Wold Eide, Eilif Solberg, Ingebjørg Kåsen

Keywords: GAN, generative adversarial networks, generative model, image synthesis, sample weighting, importance weighting, cost function, loss, mode collapse, mode dropping, coverage, divergence, FID, training dynamics, NS-GAN, MM-GAN, non-saturating, minimax

Abstract: Generative adversarial networks were introduced with a logistic MiniMax cost formulation, which normally fails to train due to saturation, and a Non-Saturating reformulation. While addressing the saturation problem, NS-GAN also inverts the generator's sample weighting, implicitly shifting emphasis from higher-scoring to lower-scoring samples when updating parameters. We present both theory and empirical results suggesting that this makes NS-GAN prone to mode dropping. We design MM-nsat, which preserves MM-GAN sample weighting while avoiding saturation by rescaling the MM-GAN minibatch gradient such that its magnitude approximates NS-GAN's gradient magnitude. MM-nsat has qualitatively different training dynamics, and on MNIST and CIFAR-10 it is stronger in terms of mode coverage, stability and FID. While the empirical results for MM-nsat are promising and favorable also in comparison with the LS-GAN and Hinge-GAN formulations, our main contribution is to show how and why NS-GAN's sample weighting causes mode dropping and training collapse.

One-sentence Summary: NS-GAN sample weighting causes mode collapse: MM-GAN has much better training dynamics, but requires a gradient rescaling to avoid saturation.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2010.02035/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=y3KSTnyQ-x