Second-Moment Loss: A Novel Regression Objective for Improved Uncertainties

Joachim Sicking, Maram Akila, Maximilian Alexander Pintz, Tim Wirtz, Asja Fischer, Stefan Wrobel

Keywords: regression, uncertainty quantification, uncertainty evaluation, dropout

Abstract: Quantification of uncertainty is one of the most promising approaches to establish safe machine learning. Despite its importance, it is far from being generally solved, especially for neural networks. One of the most commonly used approaches so far is Monte Carlo dropout, which is computationally cheap and easy to apply in practice. However, it can underestimate the uncertainty. We propose a new objective, referred to as second-moment loss (SML), to address this issue. While the full network is encouraged to model the mean, the dropout networks are explicitly used to optimize the model variance. We analyze the performance of the new objective on various toy and UCI regression datasets. Comparing to the state-of-the-art of deep ensembles, SML leads to comparable prediction accuracies and uncertainty estimates while only requiring a single model. Under distribution shift, we observe moderate improvements. From a safety perspective also the study of worst-case uncertainties is crucial. In this regard we improve considerably. Finally, we show that SML can be successfully applied to SqueezeDet, a modern object detection network. We improve on its uncertainty-related scores while not deteriorating regression quality. As a side result, we introduce an intuitive Wasserstein distance-based uncertainty measure that is non-saturating and thus allows to resolve quality differences between any two uncertainty estimates.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: We propose a novel regression loss for dropout networks that improves on the state-of-the-art of uncertainty quantification.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2012.12687/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=xSrvVgjU2s