Leaving Reality to Imagination: Robust Classification via Generated Datasets

Hritik Bansal, Aditya Grover

Keywords: Robust, Classification, Generated Data, Stable Diffusion, Generative Modeling, ImageNet-G

TL;DR: We find that Imagenet classifiers trained on real data augmented with generated data achieve high accuracy on natural distribution shifts. We further propose ImageNet-G, an evolving dataset to aid research in robust and trustworthy machine learning.

Abstract: Recent research on robustness has revealed significant performance gaps between neural image classifiers trained on datasets that are similar to the test set, and those that are from a naturally shifted distribution, such as sketches, paintings, and animations of the object categories observed during training. Prior work focuses on reducing this gap by designing engineered augmentations of training data or through unsupervised pretraining of a single large model on massive in-the-wild training datasets scraped from the Internet. However, the notion of a dataset is also undergoing a paradigm shift in recent years. With drastic improvements in the quality, ease-of-use, and access to modern generative models, generated data is pervading the web. In this light, we study the question: How do these generated datasets influence the natural robustness of image classifiers? We find that Imagenet classifiers trained on real data augmented with generated data achieve higher accuracy and effective robustness than standard training and popular augmentation strategies in the presence of natural distribution shifts. Further, we introduce and analyze an evolving generated dataset, ImageNet-G-v1, to better benchmark the design, utility, and critique of standalone generated datasets for robust and trustworthy machine learning.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2302.02503/code)