A Large-scale Study on Training Sample Memorization in Generative Modeling

Ching-Yuan Bai, Hsuan-Tien Lin, Colin Raffel, Wendy Kan

Keywords: GAN, generative adversarial networks, generative modeling, memorization

Abstract:

Many recent developments on generative models for natural images have relied on heuristically-motivated metrics that can be easily gamed by memorizing a small sample from the true distribution or training a model directly to improve the metric.
In this work, we critically evaluate the gameability of the benchmarking procedure by running a competition which ultimately resulted in participants attempting to cheat. Our competition received over 11000 submitted models which allowed us to investigate memorization-aware metrics for measuring generative model performance. Specifically, we propose the Memorization-Informed Frechet Inception Distance (MiFID) and discuss ways to ensure that winning submissions were based on genuine improvements in perceptual quality. We evaluate the effectiveness of our benchmark by manually inspecting the code for the 1000 top-performing models and labeling different forms of memorization that were intentionally or unintentionally used. To facilitate future work on benchmarking generative models, we release generated images and our labels for these models as well as code to compute the MiFID metric.

One-sentence Summary: We critically evaluate training sample memorization in generative modeling by running a competition and analyze the submissions, hence evaluate the effectiveness of state of the art metrics for generative modeling such as FID.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=2ehGiQJB4m