Exploring the Link Between Out-of-Distribution Detection and Conformal Prediction with Illustrations of Its Benefits

Paul Novello, Joseba Dalmau, Léo Andéol

Keywords: Out-of-distribution detection, Conformal Prediction, benchmark, nonconformity scores

TL;DR: We show that Out-of-distribution detection and Conformal Prediction are naturally intertwinded and emphasize some benefits of this link.

Abstract:

Research on Out-Of-Distribution (OOD) detection focuses mainly on building scores that efficiently distinguish OOD data from In Distribution (ID) data.
On the other hand, Conformal Prediction (CP) uses non-conformity scores to construct prediction sets with probabilistic coverage guarantees. In other words, the former designs scores, while the latter designs probabilistic guarantees based on scores. Therefore, we claim that these two fields might be naturally intertwined.
This work advocates for cross-fertilization between OOD and CP by formalizing their link and emphasizing two benefits of using them jointly.
First, we show that in standard OOD benchmark settings, evaluation metrics can be overly optimistic due to the test dataset's finite sample size.
Based on the work of (Bates et al, 2022), we define new *conformal AUROC* and *conformal FRP@TPR$\beta$* metrics,
which are corrections that provide probabilistic conservativeness guarantees on the variability of these metrics.
We show the effect of these corrections on two reference OOD and anomaly detection benchmarks, OpenOOD (Yang et al, 2022) and ADBench (Han et al. 2022).
Second, we explore using OOD scores as non-conformity scores and show that they can improve the efficiency of the prediction sets obtained with CP.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6562