Offline Deep models calibration with bayesian neural networks

Juan Maroñas, Roberto Paredes, Daniel Ramos

Abstract:

We apply Bayesian Neural Networks to improve calibration of state-of-the-art deep
neural networks. We show that, even with the most basic amortized approximate
posterior distribution, and fast fully connected neural network for the likelihood,
the Bayesian framework clearly outperforms other simple maximum likelihood
based solutions that have recently shown very good performance, as temperature
scaling. As an example, we reduce the Expected Calibration
Error (ECE) from 0.52 to 0.24 on CIFAR-10 and from 4.28 to 2.456 on CIFAR-100
on two Wide ResNet with 96.13% and 80.39% accuracy respectively, which are
among the best results published for this task. We demonstrate our robustness and
performance with experiments on a wide set of state-of-the-art computer vision
models. Moreover, our approach acts off-line, and thus can be applied to any
probabilistic model regardless of the limitations that the model may present during
training. This make it suitable to calibrate systems that make use of pre-trained
deep neural networks that are expensive to train for a specific task, or to directly
train a calibrated deep convolutional model with Monte Carlo Dropout approximations, among others. However,
our method is still complementary with any Bayesian Neural Network for further
improvement.

Keywords: calibration, deep models, bayesian neural networks

TL;DR: We apply bayesian neural networks to improve calibration