Rethinking OOD Detection at Scale through Ensemble Diversity

Alexander Rubinstein, Luca Scimeca, Damien Teney, Seong Joon Oh

Keywords: ensemble diversification, ood detection, ood generalizatin, disagreement regularizer

Abstract: The common practice of equating in-distribution (ID) data with the training set is a flawed oversimplification for large-scale applications. A viable strategy for out-of-distribution (OOD) detection is to train an ensemble of models to disagree, a method proven effective at smaller scales. However, these approaches have been limited by their reliance on external OOD datasets for diversification. This work revisits the fundamental definition of OOD data based on data density. This perspective reveals that the low-density, “OOD-like” samples required for diversification are already present within large training sets, removing the need for external data. We introduce the loss-guided diversification regulariser (LDR) to operationalise this principle. LDR identifies these internal samples by targeting those with high cross-entropy loss and encourages the ensemble to disagree only on them, thereby learning diverse, generalisable hypotheses. To ensure scalability, LDR employs a stochastic pairing strategy which reduces computational complexity from quadratic to constant. The process also yields a new uncertainty metric, the predictive diversity score (PDS). Extensive evaluation on benchmarks like ImageNet shows that LDR, combined with PDS, achieves state-of-the-art performance in OOD detection. Our work demonstrates that sourcing disagreement from within the training set is a powerful and effective paradigm for building robust models at scale.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 2591