Neural Bootstrapper

Minsuk Shin, Hyungjoo Cho, Hyun-seok Min, Sungbin Lim

Keywords: Bootstrapping, Uncertainty Estimation

TL;DR: We propose NeuBoots which efficiently generate bootstrap distributions and demonstrate its utility in bagging and various tasks related to uncertainty estimation.

Abstract:

Bootstrapping has been a primary tool for ensemble and uncertainty quantification in machine learning and statistics.
However, due to its nature of multiple training and resampling, bootstrapping deep neural networks is computationally burdensome; hence it has difficulties in practical application to the uncertainty estimation and related tasks. To overcome this computational bottleneck, we propose a novel approach called Neural Bootstrapper (NeuBoots), which learns to generate bootstrapped neural networks through single model training. NeuBoots injects the bootstrap weights into the high-level feature layers of the backbone network and outputs the bootstrapped predictions of the target, without additional parameters and the repetitive computations from scratch. We apply NeuBoots to various machine learning tasks related to uncertainty quantification, including prediction calibrations in image classification and semantic segmentation, active learning, and detection of out-of-distribution samples. Our empirical results show that NeuBoots outperforms other bagging based methods under a much lower computational cost without losing the validity of bootstrapping.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

Code: https://github.com/sungbinlim/NeuBoots

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2010.01051/code)