DIETing: Self-Supervised Learning with Instance Discrimination Learns Identifiable Features

Attila Juhos, Alice Bizeul, Patrik Reizinger, Randall Balestriero, David Klindt, Mark Ibrahim, Julia E Vogt, Wieland Brendel

Keywords: Self-supervised representation learning, Identifiability

TL;DR: We provide identiafiability guarantees for DIET, a recently proposed self-supervised learning method which relies on fewer bells and whistles than state-of-the-art SSL.

Abstract: Self-Supervised Learning (SSL) methods often consist of elaborate pipelines with hand-crafted data augmentations and computational tricks. However, it is unclear what is the provably minimal set of building blocks that ensures good downstream performance. The recently proposed instance discrimination method, coined DIET, stripped down the SSL pipeline and demonstrated how a simple SSL algorithm can work by predicting the sample index. Our work proves that DIET recovers cluster-based latent representations, while successfully identifying the correct cluster centroids in its classification head. We demonstrate the identifiability of DIET on synthetic data adhering to and violating our assumptions, revealing that the recovery of the cluster centroids is even more robust than the feature recovery.

Submission Number: 52