Improving Open-World Classification with Disentangled Foreground and Background Features

Choubo Ding, Guansong Pang

Abstract: Detecting out-of-distribution (OOD) inputs is a principal task for ensuring the safety of deploying deep-neural-network classifiers in open-world scenarios. OOD samples can be drawn from arbitrary distributions and exhibit deviations from in-distribution (ID) data in various dimensions, such as foreground features (e.g., objects in CIFAR100 images vs. that in CIFAR10 images) and background features (e.g., textural images vs. objects in CIFAR10). Existing methods can confound foreground and background features in training, failing to utilize the background features for OOD detection. This paper considers the importance of feature disentanglement in open-world classification and proposes the simultaneous exploitation of both foreground and background features to support the detection of OOD inputs in open-world classification. To this end, we propose a novel framework that first disentangles foreground and background features from ID training samples via a dense prediction approach, and then learns a new classifier that can evaluate the OOD scores of test images from both foreground and background features. It is a generic framework that allows for a seamless combination with various existing OOD detection methods. Extensive experiments show that our approach 1) can substantially enhance the performance of four different state-of-the-art (SotA) OOD detection methods on multiple widely-used OOD datasets with diverse background features, and 2) achieves new SotA performance on these benchmarks.

Primary Subject Area: [Content] Media Interpretation

Relevance To Conference: This study delves into the out-of-distribution detection problem in open-world image classification within multimedia, a critical aspect for ensuring the security of image classifiers. We observed that existing methods exhibit misconceptions in image interpretation, overlooking spurious correlations within images and the rich information present in image backgrounds. By disentangling foreground and background features to refine image interpretation, we have successfully achieved state-of-the-art performance in the out-of-distribution detection task.

Submission Number: 5051