An empirical study on evaluation metrics of generative adversarial networks

Gao Huang, Yang Yuan, Qiantong Xu, Chuan Guo, Yu Sun, Felix Wu, Kilian Weinberger

Abstract: Despite the widespread interest in generative adversarial networks (GANs), few works have studied the metrics that quantitatively evaluate GANs' performance. In this paper, we revisit several representative sample-based evaluation metrics for GANs, and address the important problem of \emph{how to evaluate the evaluation metrics}. We start with a few necessary conditions for metrics to produce meaningful scores, such as distinguishing real from generated samples, identifying mode dropping and mode collapsing, and detecting overfitting. Then with a series of carefully designed experiments, we are able to comprehensively investigate existing sample-based metrics and identify their strengths and limitations in practical settings. Based on these results, we observe that kernel Maximum Mean Discrepancy (MMD) and the 1-Nearest-Neighbour (1-NN) two-sample test seem to satisfy most of the desirable properties, provided that the distances between samples are computed in a suitable feature space. Our experiments also unveil interesting properties about the behavior of several popular GAN models, such as whether they are memorizing training samples, and how far these state-of-the-art GANs are from perfect.

Keywords: generative adversarial networks, evaluation metric

Code: [ xuqiantong/GAN-Metrics](https://github.com/xuqiantong/GAN-Metrics) + [ 3 community implementations](https://paperswithcode.com/paper/?openreview=Sy1f0e-R-)

Data: [ImageNet](https://paperswithcode.com/dataset/imagenet)

Community Implementations: [ 4 code implementations](https://www.catalyzex.com/paper/an-empirical-study-on-evaluation-metrics-of/code)