Neural Ensemble Search for Uncertainty Estimation and Dataset Shift

Sheheryar Zaidi, Arber Zela, Thomas Elsken, Chris Holmes, Frank Hutter, Yee Whye Teh

Keywords: uncertainty estimation, deep ensemble, dataset shift, robustness, uncertainty calibration

Abstract: Ensembles of neural networks achieve superior performance compared to stand-alone networks not only in terms of predictive performance, but also uncertainty calibration and robustness to dataset shift. Diversity among networks is believed to be key for building strong ensembles, but typical approaches, such as \emph{deep ensembles}, only ensemble different weight vectors of a fixed architecture. Instead, we propose two methods for constructing ensembles to exploit diversity among networks with \emph{varying} architectures. We find that the resulting ensembles are indeed more diverse and also exhibit better uncertainty calibration, predictive performance and robustness to dataset shift in comparison with deep ensembles on a variety of classification tasks.

One-sentence Summary: We propose methods for constructing ensembles of neural networks with varying architectures, demonstrating that they outperform deep ensembles, in terms of uncertainty calibration, predictive performance and robustness to dataset shift.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=fCECU-SaOj