Training independent subnetworks for robust prediction

Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshminarayanan, Andrew Mingbo Dai, Dustin Tran

Keywords: Efficient ensembles, robustness

Abstract:

Recent approaches to efficiently ensemble neural networks have shown that strong robustness and uncertainty performance can be achieved with a negligible gain in parameters over the original network. However, these methods still require multiple forward passes for prediction, leading to a significant runtime cost. In this work, we show a surprising result:
the benefits of using multiple predictions can be achieved 'for free' under a single model's forward pass. In particular, we show that, using a multi-input multi-output (MIMO) configuration, one can utilize a single model's capacity to train multiple subnetworks that independently learn the task at hand. By ensembling the predictions made by the subnetworks, we improve model robustness without increasing compute. We observe a significant improvement in negative log-likelihood, accuracy, and calibration error on CIFAR10, CIFAR100, ImageNet, and their out-of-distribution variants compared to previous methods.

One-sentence Summary: We show that a deep neural network can be trained to give multiple independent predictions simultaneously, which results in a computationally efficient ensemble model.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Code: [ google/uncertainty-baselines](https://github.com/google/uncertainty-baselines) + [ 1 community implementation](https://paperswithcode.com/paper/?openreview=OGg9XnKxFAH)

Community Implementations: [ 3 code implementations](https://www.catalyzex.com/paper/arxiv:2010.06610/code)