LevyScore: A Fast Sample-Wise Confidence Score of Pretrained Joint Embedding Model

Lucas Maes, Damien Scieur, Randall Balestriero

Track: Extended Abstract Track

Keywords: Confidence Estimation, Gaussian Embeddings, JEPA

Abstract: Foundation models--Deep Networks (DNs) able to solve numerous downstream task without requiring to be retrained--have made enormous strides in recent years. Thus far, progress has mostly been measured in terms of average performance on mostly curated datasets. Yet, a large number of end-users are concerned with sensitive applications for which an assessment of the foundation model's confidence is required. To that end, we propose LevyScore--a simple, fast sample-wise confidence score for any pretrained foundation model using joint-embeddings. LevyScore is theoretically sound as it captured the deviation of an embedding from its pretraining density. Yet, LevyScore does not require knowledge of the pretraining data nor having access to any downstream dataset. Instead it is built from a core principle of Joint Embeddings: producing Gaussian embeddings. Our experiments demonstrate that LevyScore provides an effective mechanism for filtering samples according to the foundation model’s confidence. Across probes and datasets, it consistently improves the accuracy--coverage tradeoff, achieving state-of-the-art performance. By selectively discarding uncertain predictions, LevyScore offers a simple, principled, and practical tool for deploying foundation models in high-stakes applications.

Submission Number: 119