Are all negatives created equal in contrastive instance discrimination?

Tiffany Cai, Jonathan Frankle, David J. Schwab, Ari S. Morcos

Keywords: self-supervised learning, contrastive learning, contrastive instance discrimination, negatives, understanding self-supervised learning, ssl

Abstract: Self-supervised learning has recently begun to rival supervised learning on computer vision tasks. Many of the recent approaches have been based on contrastive instance discrimination (CID), in which the network is trained to recognize two augmented versions of the same instance (a query and positive while discriminating against a pool of other instances (negatives). Using MoCo v2 as our testbed, we divided negatives by their difficulty for a given query and studied which difficulty ranges were most important for learning useful representations. We found that a small minority of negatives--just the hardest 5%--were both necessary and sufficient for the downstream task to reach full accuracy. Conversely, the easiest 95% of negatives were unnecessary and insufficient. Moreover, we found that the very hardest 0.1% of negatives were not only unnecessary but also detrimental. Finally, we studied the properties of negatives that affect their hardness, and found that hard negatives were more semantically similar to the query, and that some negatives were more consistently easy or hard than we would expect by chance. Together, our results indicate that negatives play heterogeneous roles and CID may benefit from more intelligent negative treatment.

One-sentence Summary: We study the relative importance of negatives in contrastive instance discrimination, finding that only the hardest 5% of negatives are necessary and sufficient for good performance, and these negatives are more semantically similar to the query.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=yjrtEuEAun