Evaluation of generative networks through their data augmentation capacity

Timothée Lesort, Florian Bordes, Jean-Francois Goudou, David Filliat

Abstract: Generative networks are known to be difficult to assess. Recent works on generative models, especially on generative adversarial networks, produce nice samples of varied categories of images. But the validation of their quality is highly dependent on the method used. A good generator should generate data which contain meaningful and varied information and that fit the distribution of a dataset. This paper presents a new method to assess a generator. Our approach is based on training a classifier with a mixture of real and generated samples. We train a generative model over a labeled training set, then we use this generative model to sample new data points that we mix with the original training data. This mixture of real and generated data is thus used to train a classifier which is afterwards tested on a given labeled test dataset. We compare this result with the score of the same classifier trained on the real training data mixed with noise. By computing the classifier's accuracy with different ratios of samples from both distributions (real and generated) we are able to estimate if the generator successfully fits and is able to generalize the distribution of the dataset. Our experiments compare the result of different generators from the VAE and GAN framework on MNIST and fashion MNIST dataset.

TL;DR: Evaluating generative networks through their data augmentation capacity on discrimative models.

Keywords: Generative models, Evaluation of generative models, Data Augmentation

Data: [Fashion-MNIST](https://paperswithcode.com/dataset/fashion-mnist)