Understanding and Stabilizing GANs' Training Dynamics with Control Theory

Kun Xu, Chongxuan Li, Huanshu Wei, Jun Zhu, Bo Zhang

Keywords: Generative Adversarial Nets, Stability Analysis, Control Theory

TL;DR: We adopt the control theory to understand and stabilize the dynamics of GANs.

Abstract: Generative adversarial networks~(GANs) have made significant progress on realistic image generation but often suffer from instability during the training process. Most previous analyses mainly focus on the equilibrium that GANs achieve, whereas a gap exists between such theoretical analyses and practical implementations, where it is the training dynamics that plays a vital role in the convergence and stability of GANs. In this paper, we directly model the dynamics of GANs and adopt the control theory to understand and stabilize it. Specifically, we interpret the training process of various GANs as certain types of dynamics in a unified perspective of control theory which enables us to model the stability and convergence easily. Borrowed from control theory, we adopt the widely-used negative feedback control to stabilize the training dynamics, which can be considered as an $L2$ regularization on the output of the discriminator. We empirically verify our method on both synthetic data and natural image datasets. The results demonstrate that our method can stabilize the training dynamics as well as converge better than baselines.

Code: https://anonymous.4open.science/r/a5e02628-d2d6-43b4-b645-5752fb87637a/

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:1909.13188/code)