Scalable Ensemble Diversification for OOD Generalization and Detection

Alexander Rubinstein, Luca Scimeca, Damien Teney, Seong Joon Oh

Keywords: diverse, ensemble, scalable, robustness, uncertainty, OOD detection, OOD generalization

Abstract:

Training a diverse ensemble of models has several practical applications such as providing candidates for model selection with better out-of-distribution (OOD) generalization, and enabling the detection of OOD samples via Bayesian principles.
An existing approach to diverse ensemble training encourages the models to disagree on provided OOD samples. However, the approach is computationally expensive and it requires well-separated ID and OOD examples, such that it has only been demonstrated in small-scale settings.

**Method.**
This work presents a Hardness-based Diversification Regularizer (HDR)
applicable to large-scale settings (e.g. ImageNet)
that does not require OOD samples.
Instead, HDR identifies hard training samples on the fly
and encourages the ensemble members to disagree on these.
To improve scaling, we show how to avoid the expensive computations in existing methods of exhaustive pairwise disagreements across models.

**Results.**
We evaluate the benefits of diversification with experiments on ImageNet.
First, for OOD generalization, we observe large benefits from the diversification in multiple settings including output-space (classical) ensembles and weight-space ensembles (model soups).
Second, for OOD detection, we turn the diversity of ensemble hypotheses into a novel uncertainty score estimator that surpasses a large number of OOD detection baselines.

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 448