Optimal Zero-shot Regret Minimization for Selective Classification with Out-of-Distribution Detection

Eduardo Dadalto Câmara Gomes, Marco Romanelli

Keywords: Trustworthy AI, Selective Classification, Out-of-Distribution Detection, Information Theory

TL;DR: We propose a novel zero-shot black-box method for Selective Classification with Out-of-Distribution Detection (SCOD) that aggregates off-the-shelf detectors, is optimal for regret minimization, and provides guarantees on the worst-case performance.

Abstract: Selective Classification with Out-of-Distribution Detection (SCOD) is a general framework that combines the detection of incorrectly classified in-distribution samples and out-of-distribution samples. Previous solutions for SCOD heavily rely on the choice of Selective Classification (SC) and Out-of-Distribution (OOD) detectors selected at test time. Notably, the performance of these detectors varies across different underlying data distributions. Hence, a poor choice can affect the efficacy of the SCOD framework. On the other hand, making an informed choice is impossible without samples from both in- and out-distribution. We propose an optimal zero-shot black-box method for SCOD that aggregates off-the-shelf detectors, is based on the principle of regret minimization, and therefore provides guarantees on the worst-case performance. We demonstrate that our method achieves performance comparable to state-of-the-art methods in several benchmarks while also shielding the user from the burden of blindly selecting the SC and OOD detectors, optimally reducing the worst-case rejection risk.

Readers: auai.org/UAI/2025/Conference, auai.org/UAI/2025/Conference/Area_Chairs, auai.org/UAI/2025/Conference/Reviewers, auai.org/UAI/2025/Conference/Submission147/Authors, auai.org/UAI/2025/Conference/Submission147/Reproducibility_Reviewers

Submission Number: 147