Understanding Negative Samples in Instance Discriminative Self-supervised Representation LearningDownload PDF

May 21, 2021 (edited Jan 14, 2022)NeurIPS 2021 PosterReaders: Everyone
  • Keywords: representation learning, self-supervised representation learning
  • TL;DR: We give a theoretical analysis to explain a benefit of large negative samples in self-supervised contrastive learning.
  • Abstract: Instance discriminative self-supervised representation learning has been attracted attention thanks to its unsupervised nature and informative feature representation for downstream tasks. In practice, it commonly uses a larger number of negative samples than the number of supervised classes. However, there is an inconsistency in the existing analysis; theoretically, a large number of negative samples degrade classification performance on a downstream supervised task, while empirically, they improve the performance. We provide a novel framework to analyze this empirical result regarding negative samples using the coupon collector's problem. Our bound can implicitly incorporate the supervised loss of the downstream task in the self-supervised loss by increasing the number of negative samples. We confirm that our proposed analysis holds on real-world benchmark datasets.
  • Supplementary Material: pdf
  • Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.
  • Code: https://github.com/nzw0301/Understanding-Negative-Samples
17 Replies