Fine-Grained Uncertainty Decomposition in Large Language Models: A Spectral Approach

Nassim Walha, Sebastian G. Gruber, Thomas Decker, Yinchong Yang, Alireza Javanmardi, Eyke Hüllermeier, Florian Buettner

Keywords: Large Language Models, LLMs, Uncertainty Quantification, Reliability, Reliable ML, Reliable AI, Aleatoric Uncertainty, Epistemic Uncertainty, Spectral Uncertainty, Von Neumann Entropy, Semantic Similarity, Natural Language Processing, NLP, Predictive Uncertainty, Semantic Ambiguity, Model Confidence, Uncertainty Decomposition.

TL;DR: We propose Spectral Uncertainty, a theoretically-derived method to accurately separate and measure different fine-grained uncertainty types in Large Language Models.

Abstract: As Large Language Models (LLMs) are increasingly integrated in diverse applications, obtaining reliable measures of their predictive uncertainty has become critically important. A precise distinction between aleatoric uncertainty, arising from inherent ambiguities within input data, and epistemic uncertainty, originating exclusively from model limitations, is essential to effectively address each uncertainty source and improve the reliability of the user-LLM interaction. In this paper, we introduce Spectral Uncertainty, a novel approach to quantifying and decomposing uncertainties in LLMs. Leveraging the Von Neumann entropy from quantum information theory, Spectral Uncertainty provides a rigorous theoretical foundation for separating total uncertainty into distinct aleatoric and epistemic components. Unlike existing baseline methods, our approach incorporates a fine-grained representation of semantic similarity, enabling nuanced differentiation among various semantic interpretations in model responses. Empirical evaluations demonstrate that Spectral Uncertainty outperforms state-of-the-art methods in estimating both aleatoric and total uncertainty across diverse models and benchmark datasets.

Submission Number: 103