In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation

Julian Bitterwolf, Maximilian Mueller, Matthias Hein

Keywords: out-of-distribution detection, dataset reliability, robustness, trustworthiness of evaluation data, new datasets

TL;DR: Commonly used OOD test datasets contain large fractions of in-distribution data, resulting in limited trustworthiness of OOD detection evaluations. We introduce the NINCO OOD dataset, each sample of which we checked to be truly OOD.

Abstract: Out-of-distribution (OOD) detection is the problem of identifying inputs which are unrelated to the in-distribution task. The OOD detection performance when the in-distribution (ID) is ImageNet-1K is commonly being tested on a small range of test OOD datasets. We find that most of the currently used test OOD datasets have severe issues, in some cases more than 50% of the dataset contains objects belonging to one of the ID classes. These erroneous samples heavily distort the evaluation of OOD detectors. As a solution, we introduce with NINCO a novel test OOD dataset, each sample checked to be ID free, which with its fine-grained range of OOD classes allows for a detailed analysis of an OOD detector’s strengths and failure modes, particularly when paired with a number of synthetic “OOD unit-tests”. We provide detailed evaluations across a large set of architectures and OOD detection methods on NINCO and the unit-tests, revealing new insights about model weaknesses and the effects of pretraining on OOD detection performance.

Community Implementations: [ 5 code implementations](https://www.catalyzex.com/paper/arxiv:2306.00826/code)