Scalable Ensemble Diversification for OOD Generalization and Detection

Alexander Rubinstein, Luca Scimeca, Damien Teney, Seong Joon Oh

Keywords: diverse, ensemble, scalable, robustness, uncertainty, OOD detection, OOD generalization

Abstract: Training a diverse ensemble of models has several practical application scenarios, such as model selection for out-of-distribution (OOD) generalization and the detection of OOD samples via Bayesian principles. Previous approaches to diverse ensemble training have relied on the framework of letting the models make the correct predictions for the given in-distribution (ID) data while letting them come up with different hypotheses for the OOD data. As such, they require well-separated ID and OOD datasets to ensure a performant and diverse ensemble and have only been verified in smaller-scale lab environments where such a separation is readily available. In this work, we propose a framework, Scalable Ensemble Diversification (SED), for scaling up existing diversification methods to large-scale datasets and tasks (e.g. ImageNet), where the ID-OOD separation may not be available. SED automatically identifies OOD samples within the large-scale ID dataset on the fly and encourages the ensemble to make diverse hypotheses on them. To make SED more suitable for large-scale applications, we propose an algorithm to speed up the expensive pairwise disagreement computation. We verify the resulting diversification of the ensemble on ImageNet and demonstrate the benefit of diversification on the OOD generalization and OOD detection tasks. In particular, for OOD detection, we propose a novel uncertainty score estimator based on the diversity of ensemble hypotheses, which lets SED surpass all the considered baselines in OOD detection task. Code will be available soon.

Primary Area: Machine vision

Submission Number: 1317