Combining Ensembles and Data Augmentation Can Harm Your Calibration
Keywords: Ensembles, Uncertainty estimates, Calibration
Abstract: Ensemble methods which average over multiple neural network predictions are a simple approach to improve a model’s calibration and robustness. Similarly, data augmentation techniques, which encode prior information in the form of invariant feature transformations, are effective for improving calibration and robustness. In this paper, we show a surprising pathology: combining ensembles and data augmentation can harm model calibration. This leads to a trade-off in practice, whereby improved accuracy by combining the two techniques comes at the expense of calibration. On the other hand, selecting only one of the techniques ensures good uncertainty estimates at the expense of accuracy. We investigate this pathology and identify a compounding under-confidence among methods which marginalize over sets of weights and data augmentation techniques which soften labels. Finally, we propose a simple correction, achieving the best of both worlds with significant accuracy and calibration gains over using only ensembles or data augmentation individually. Applying the correction produces new state-of-the art in uncertainty calibration and robustness across CIFAR-10, CIFAR-100, and ImageNet.
One-sentence Summary: We found that combining ensembles and data augmentation worsens calibration than applying them individually, and we proposed a simple fix to it.
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics
Data: [CIFAR-10](https://paperswithcode.com/dataset/cifar-10), [CIFAR-100](https://paperswithcode.com/dataset/cifar-100), [CIFAR-10C](https://paperswithcode.com/dataset/cifar-10c), [ImageNet](https://paperswithcode.com/dataset/imagenet)
Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2010.09875/code)
16 Replies
Loading