Pointwise Information Measures as Confidence Estimators in Deep Neural Networks: A Comparative Study

Shelvia Wongso, Rohan Ghosh, Mehul Motani

TL;DR: We explore how information-theoretic measures can improve confidence estimation in neural networks.

Abstract: Estimating the confidence of deep neural network predictions is crucial for safe deployment in high-stakes applications. While softmax probabilities are commonly used, they are often poorly calibrated, and existing calibration methods have been shown to be detrimental to failure prediction. In this paper, we propose using information-theoretic measures to estimate prediction confidence in a post-hoc manner, without modifying network architecture or training. Specifically, we compare three pointwise information (PI) measures: pointwise mutual information (PMI), pointwise $\mathcal{V}$-information (PVI), and the recently proposed pointwise sliced mutual information (PSI). These measures are theoretically grounded in their relevance to predictive uncertainty, with properties such as invariance, convergence rates, and sensitivity to geometric attributes like margin and intrinsic dimensionality. Through extensive experiments on benchmark computer vision models and datasets, we find that PVI consistently outperforms PMI, PSI and existing post-hoc baselines in failure prediction across metrics. For confidence calibration, PVI matches the performance of temperature-scaled softmax, which is already regarded as a highly effective baseline. This indicates that PVI achieves superior failure prediction without compromising its calibration performance. This aligns with our theoretical insights, which suggest that PVI offers the most balanced trade-offs.

Lay Summary: Deep neural networks often produce overconfident predictions, which can be dangerous in high-stakes applications like healthcare or autonomous driving. While softmax probabilities are commonly used to estimate confidence, they are often poorly calibrated, and many existing methods actually make failure prediction worse. In this work, we explore three information-theoretic measures: (1) Pointwise Mutual Information (PMI), (2) Pointwise V-Information (PVI), and (3) Pointwise Sliced Mutual Information (PSI), as post-hoc tools for estimating confidence, without changing the model’s architecture or training. We analyze how these measures behave under different data transformations and geometric properties like margin and dimensionality. Our experiments on standard computer vision benchmarks show that PVI consistently outperforms PMI, PSI, and existing confidence estimation methods. These findings suggest that PVI offers a more balanced and reliable way to assess model confidence, especially for detecting when the model is likely to fail.

Link To Code: https://github.com/kentridgeai/PI-Conf-Est

Primary Area: Social Aspects->Accountability, Transparency, and Interpretability

Keywords: information theory, confidence estimation, deep neural networks

Submission Number: 14884