Practical Evaluation of Out-of-Distribution Detection Methods for Image Classification

Engkarat Techapanurak, Takayuki Okatani

Keywords: out-of-distribution, novel class detection, domain shift, concept drift

Abstract: We reconsider the evaluation of OOD detection methods for image recognition. Although many studies have been conducted so far to build better OOD detection methods, most of them follow Hendrycks and Gimpel's work for the method of experimental evaluation. While the unified evaluation method is necessary for a fair comparison, there is a question of if its choice of tasks and datasets reflect real-world applications and if the evaluation results can generalize to other OOD detection application scenarios. In this paper, we experimentally evaluate the performance of representative OOD detection methods for three scenarios, i.e., irrelevant input detection, novel class detection, and domain shift detection, on various datasets and classification tasks. The results show that differences in scenarios and datasets alter the relative performance among the methods. Our results can also be used as a guide for practitioners for the selection of OOD detection methods.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: This paper provides a practical evaluation of OOD detection methods that is missing in previous studies and will serve as a practitioners' guide.

Reviewed Version (pdf): https://openreview.net/references/pdf?id=KlbBHL5Hw_