Towards More Realistic Neural Network Uncertainties

Joachim Sicking, Alexander Kister, Matthias Fahrland, Stefan Eickeler, Fabian Hueger, Stefan Rueping, Peter Schlicht, Tim Wirtz

Abstract: Statistical models are inherently uncertain. Quantifying or at least upper-bounding their uncertainties is vital for safety-critical systems. While standard neural networks do not report this information, several approaches exist to integrate uncertainty estimates into them. Assessing the quality of these uncertainty estimates is not straightforward, as no direct ground truth labels are available. Instead, implicit statistical assessments are required. For regression, we propose to evaluate uncertainty realism---a strict quality criterion---with a Mahalanobis distance-based statistical test. An empirical evaluation reveals the need for uncertainty measures that are appropriate to upper-bound heavy-tailed empirical errors. Alongside, we transfer the variational U-Net classification architecture to standard supervised image-to-image tasks. It provides two uncertainty mechanisms and significantly improves uncertainty realism compared to a plain encoder-decoder model.

Keywords: uncertainty, variational inference, MC dropout, variational autoencoder, evaluation

TL;DR: We assess and improve the quality of neural network uncertainties by proposing an evaluation criterion and introducing a new uncertainty mechanism.