Internet Curiosity: Directed Unsupervised Learning on Uncurated Internet Data

Alexander C. Li, Ellis Brown, Alexei A. Efros, Deepak Pathak

Abstract: We show that a curiosity-driven computer vision algorithm can learn to efficiently query Internet text-to-image search engines for images that improve the model’s performance on a specified dataset. In contrast to typical self-supervised computer vision algorithms, which learn from static datasets, our model actively expands its training set with the most relevant images. First, we calculate an image-level curiosity reward that encourages our model to find the most useful images for pre-training. Second, we use text similarity scores to propagate observed curiosity rewards to untried text queries. This efficiently identifies relevant semantic clusters without any need for class labels or label names from the targeted dataset. Our method significantly outperforms models that require 1–2 orders of magnitude more compute and data.