Learning to See by Looking at Noise

Manel Baradad, Jonas Wulff, Tongzhou Wang, Phillip Isola, Antonio Torralba

Keywords: unsupervised representation learning, image generation, contrastive training, learning from noise

TL;DR: We study unsupervised representation learning by training with generative image models that do not use real data.

Abstract:

Current vision systems are trained on huge datasets, and these datasets come with costs: curation is expensive, they inherit human biases, and there are concerns over privacy and usage rights. To counter these costs, interest has surged in learning from cheaper data sources, such as unlabeled images. In this paper we go a step further and ask if we can do away with real image datasets entirely, instead learning from procedural noise processes.
We investigate a suite of image generation models that produce images from simple random processes. These are then used as training data for a visual representation learner with a contrastive loss. In particular, we study statistical image models, randomly initialized deep generative models, and procedural graphics models.
Our findings show that it is important for the noise to capture certain structural properties of real data but that good performance can be achieved even with processes that are far from realistic. We also find that diversity is a key property to learn good representations.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

Code: https://github.com/mbaradad/learning_with_noise

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2106.05963/code)