How and Why We Detect Distribution Shift: Critical Analysis of Methods and Benchmarks

Hongjun Wang, Sagar Vaze, Kai Han

Keywords: Open-set Recognition, Out of distribution Detection

TL;DR: we aim to provide a consolidated view of the two largest sub-fields: open-set recognition (OSR) and out-of-distribution detection (OOD)

Abstract:

Detecting test-time distribution shift has emerged as a key capability for safely deployed machine learning models, with the question being tackled under various guises in recent years. In this paper, we aim to provide a consolidated view of the two largest sub-fields within the community: open-set recognition (OSR) and out-of-distribution detection (OOD). In particular, we aim to provide rigorous empirical analysis of different methods across settings and provide actionable takeaways for practitioners and researchers. Concretely, we make the following contributions:
(i) For the first time, we perform rigorous cross-evaluation between state-of-the-art methods in the OOD and OSR settings and identify a strong correlation between the performances of methods for them;
(ii) We propose a new, large-scale benchmark setting which we suggest better disentangles the problem tackled by OOD and OSR;
(iii) We thoroughly examine SOTA methods for OOD and OSR on our large-scale benchmark;
and (iv) Finally, we find that the best performing method on previous benchmarks struggles on our large-scale benchmark, while magnitude-aware scoring rules consistently show promise.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Deep Learning and representational learning