Strengthen Out-of-Distribution Detection with Uncertainty-Driven Adaptively Rectified Backpropagation

Rundong He, Jieming Shi

Keywords: Out-of-distribution detection, Uncertainty

TL;DR: We propose Adaptively Rectified Backpropagation (ARB), is a data-centric approach that prevents overfitting for OOD detection.

Abstract: Out-of-distribution (OOD) detection aims to ensure AI system reliability by detecting inputs outside the training distribution. Recent work shows that overfitting during later stages of training can hurt OOD detection. To overcome overfitting, several methods attempt to distill the model after training or prune the model during training from a model-centric perspective. In contrast, this paper proposes a data-centric end-to-end solution called Uncertainty-driven Adaptively Rectified Backpropagation (UARB), which follows the principle that once the model has mastered an instance, training on it should stop to prevent overfitting. UARB considers an instance mastered if the zero-order and second-order differences of its uncertainty value remain within a small range around zero, offering a more consistent measure of an instance’s learning status. Additionally, since different classes exhibit varying optimization progress, using a fixed threshold to determine when to exclude an instance from backpropagation is theoretically unsound. UARB develops an adaptive threshold by incorporating class-informed statistics to determine when to exclude an instance. Extensive experiments demonstrate that UARB can enhance OOD detection performance.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 10206