Lightning-UQ-Box: A Comprehensive Framework for Uncertainty Quantification in Deep Learning

Nils Lehmann, Jakob Gawlikowski, Adam J Stewart, Vytautas Jančauskas, Stefan Depeweg, Eric Nalisnick, Nina Maria Gottschling

Keywords: Uncertainty Quantification, Bayesian Deep Learning, Conformal Prediction

Abstract:

Uncertainty quantification (UQ) is an essential tool for applying deep neural networks (DNNs) to real world tasks, as it attaches a degree of confidence to DNN outputs. However, despite its benefits, UQ is often left out of the standard DNN workflow due to the additional technical knowledge required to apply and evaluate existing UQ procedures. Hence there is a need for a comprehensive toolbox that allows the user to integrate UQ into their modeling workflow, without significant overhead. We introduce Lightning UQ Box: a unified interface for applying and evaluating various approaches to UQ. In this paper, we provide a theoretical and quantitative comparison of the wide range of state-of-the-art UQ methods implemented in our toolbox. We focus on two challenging vision tasks: (i) estimating tropical cyclone wind speeds from infrared satellite imagery and (ii) estimating the power output of solar panels from RGB images of the sky. Our results demonstrate the need for a broad and approachable experimental framework for UQ, that can be used for benchmarking by highlighting the differences between UQ methods.
The toolbox, example implementations, and further information are available at: https://github.com/lightning-uq-box/lightning-uq-box.

Submission Number: 1043