Deep Ensembles Work, But Are They Necessary?

Taiga Abe, E. Kelly Buchanan, Geoff Pleiss, Richard Zemel, John Patrick Cunningham

Keywords: Deep Ensembles, Uncertainty Quantification, Robustness, Dataset Shift

TL;DR: Many of the benefits of deep ensembles to uncertainty quantification can be explained or replicated by the appropriate choice of a single neural network.

Abstract: Ensembling neural networks is an effective way to increase accuracy, and can often match the performance of individual larger models. This observation poses a natural question: given the choice between a deep ensemble and a single neural network with similar accuracy, is one preferable over the other? Recent work suggests that deep ensembles may offer distinct benefits beyond predictive power: namely, uncertainty quantification and robustness to dataset shift. In this work, we demonstrate limitations to these purported benefits, and show that a single (but larger) neural network can replicate these qualities. First, we show that ensemble diversity, by any metric, does not meaningfully contribute to an ensemble's ability to detect out-of-distribution (OOD) data, but is instead highly correlated with the relative improvement of a single larger model. Second, we show that the OOD performance afforded by ensembles is strongly determined by their in-distribution (InD) performance, and - in this sense - is not indicative of any "effective robustness." While deep ensembles are a practical way to achieve improvements to predictive power, uncertainty quantification, and robustness, our results show that these improvements can be replicated by a (larger) single model.

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2202.06985/code)