Open Peer Review. Open Publishing. Open Access. Open Discussion. Open Directory. Open Recommendations. Open API. Open Source.
Evaluating Implicit Generative Models With Large Samples
Ishaan Gulrajani, Colin Raffel, Luke Metz
Feb 12, 2018 (modified: Feb 12, 2018)ICLR 2018 Workshop Submissionreaders: everyone
Abstract:We study the problem of evaluating a generative model using only a finite sample from the model. For many common evaluation functions, generalization is meaningless because trivially memorizing the training set attains a better score than the models we consider state-of-the-art. We clarify a necessary condition for an evaluation function not to behave this way: estimating the function must require a large sample from the model. In search of such a function, we turn to parametric adversarial divergences, which are defined in terms of a neural network trained to distinguish between distributions: as we make the network larger, the function is less easily minimized by memorizing the training set. We implement a reliable evaluation function based on these ideas, validate it experimentally, and show models which achieve better scores than memorizing the training set.
TL;DR:Evaluating generalization in implicit generative models by considering millions of sample points rather than thousands
Enter your feedback below and we'll get back to you as soon as possible.