(Implicit) Ensembles of Ensembles: Epistemic Uncertainty Collapse in Large Models
Keywords: epistemic uncertainty, uncertainty quantification, exploratory research
TL;DR: Larger deep learning models unexpectedly exhibit collapsed epistemic uncertainty due to implicit ensembling, challenging conventional wisdom about the benefits of model complexity for uncertainty quantification.
Abstract: We uncover a paradoxical phenomenon in deep learning models: as model complexity increases, epistemic uncertainty often collapses, challenging the assumption that larger models invariably offer better uncertainty quantification. We propose that this collapse stems from implicit ensembling within large models. To support this hypothesis, we offer two lines of evidence: first, we demonstrate the epistemic uncertainty collapse empirically across various architectures, from explicit hierarchical ensembles and simple MLPs to state-of-the-art vision models; second, we introduce implicit ensemble extraction, a technique that decomposes larger models into diverse sub-models. We provide theoretical justification for these phenomena and explore their implications for uncertainty estimation.
Submission Number: 28
Loading