Experimental methodology to evaluate the effectiveness of uncertainty disentanglement on regression models

Kevin Pasini, Clement Arlotti, Milad Leyli-abadi, Marc Nabhan, Johanna Baro

Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Machine learning; Uncertainty quantification; Uncertainty decomposition; benchmark; Regression

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: This paper proposes an experimental evaluation methodology and carries out a benchmark for 4 uncertainty quantification approaches tackling the effectiveness of their uncertainty disentanglement.

Abstract: The lack of an acceptable confidence level associated with the predictions of Machine Learning (ML) models may inhibit their deployment and usage. A practical way to avoid this drawback is to enhance these predictions with trustworthiness and risk-aware add-ons such as Uncertainty Quantification (UQ). Typically, the quantified uncertainty mainly captures two intertwined parts: an epistemic uncertainty component linked to a lack of observed data and an aleatoric uncertainty component due to irreducible variability. Several existing UQ-paradigms aim to disentangle the total quantified uncertainty into these two parts, with the aim of distinguishing model irrelevance from high uncertainty-level decisions. However, few of them are delving deeper into evaluating the disentanglement result, even less on real-world data. In this paper, we propose and implement a methodology to assess the effectiveness of uncertainty disentanglement through benchmarking of various UQ approaches. We introduce some indicators that allow us to robustly assess the decomposition feasibility in the absence of ground truth. The evaluation is done using an epistemic variability injection mechanism on four state-of-the-art UQ approaches based on ML models, on both synthetic and real-world gas demand datasets. The obtained results show the effectiveness of the proposed methodology for better understanding and selection of the relevant UQ approach. The corresponding code and data can be found in the Github repository.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3294