LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures

Vimal Thilak, Chen Huang, Omid Saremi, Laurent Dinh, Hanlin Goh, Preetum Nakkiran, Joshua M. Susskind, Etai Littwin

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Self Supervised Learning, Joint Embedding Architectures

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: A SOTA method for predicting linear probing performance in joint embedding architectures

Abstract:

Joint embedding (JE) architectures have emerged as a promising avenue for ac-
quiring transferable data representations. A key obstacle to using JE methods,
however, is the inherent challenge of evaluating learned representations without
access to a downstream task, and an annotated dataset. Without efficient and re-
liable evaluation, it is difficult to iterate on architectural and training choices for
JE methods. In this paper, we introduce LiDAR (Linear Discriminant Analysis
Rank), a metric designed to measure the quality of representations within JE archi-
tectures. Our metric addresses several shortcomings of recent approaches based
on feature covariance rank by discriminating between informative and uninforma-
tive features. In essence, LiDAR quantifies the rank of the Linear Discriminant
Analysis (LDA) matrix associated with the surrogate SSL task—a measure that
intuitively captures the information content as it pertains to solving the SSL task.
We empirically demonstrate that LiDAR significantly surpasses naive rank based
approaches in its predictive power of optimal hyperparameters. Our proposed cri-
terion presents a more robust and intuitive means of assessing the quality of rep-
resentations within JE architectures, which we hope facilitates broader adoption
of these powerful techniques in various domains.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 8433