everyone
since 09 May 2025">EveryoneRevisionsBibTeXCC BY 4.0
In recent times, there has been a surge in the utilization of large language models (LLMs) owing to their remarkable text generation capabilities. Nevertheless, a notable concern arises from their tendency to make confident yet inaccurate predictions, emphasizing the need for assessing uncertainty in LLMs. Various methods for estimating uncertainty have been proposed in recent studies, leveraging the token probability of the model's predictions. However, the correlation and distinction between methods across different categories warrant further investigation. This study delves into the fundamental design choices of current uncertainty estimation methods and introduces a unified framework for assessing uncertainty in large language models. The primary insights of this research indicate that uncertainty information is distributed among the tokens, and the model's confidence in its uncertainty increases post-prediction. Furthermore, we introduce a novel lightweight supervised method named Adaptive Uncertainty Probing (AUP), which significantly outperforms existing methods. Through extensive experimentation, we demonstrate the efficacy, versatility, and efficiency of AUP.