Towards a Testable Notion of Generalization for Generative Adversarial Networks

Robert Cornish, Hongseok Yang, Frank Wood

Abstract: We consider the question of how to assess generative adversarial networks, in particular with respect to whether or not they generalise beyond memorising the training data. We propose a simple procedure for assessing generative adversarial network performance based on a principled consideration of what the actual goal of generalisation is. Our approach involves using a test set to estimate the Wasserstein distance between the generative distribution produced by our procedure, and the underlying data distribution. We use this procedure to assess the performance of several modern generative adversarial network architectures. We find that this procedure is sensitive to the choice of ground metric on the underlying data space, and suggest a choice of ground metric that substantially improves performance. We finally suggest that attending to the ground metric used in Wasserstein generative adversarial network training may be fruitful, and outline a concrete pathway towards doing so.

TL;DR: Assess whether or not your GAN is actually doing something other than memorizing the training data.

Keywords: generative adversarial networks, Wasserstein, GAN, generalization, theory