Perceptions of Linguistic Uncertainty by Language Models and Humans

Catarina G Belém

Abstract: Uncertainty expressions such as 'probably' or 'highly unlikely' are pervasive in human language. While prior work has established that there is general population-level agreement among humans about what these expressions mean quantitatively, the abilities of LLMs to interpret these phrases have seen little investigation. In this paper, we introduce a task for evaluating the abilities of LLMs to interpret uncertainty expressions as probabilities. Our approach assesses whether LLMs can employ theory of mind in this setting: understanding the uncertainty of another agent about a particular statement, independently of the LLM's own certainty about that statement. We evaluate both humans and a variety of LLMs on this task, demonstrating that a variety of LLMs are able to map uncertainty expressions to probabilistic responses in a human-like manner. However, we observe systematically different behavior depending on whether a statement is actually true or false. This sensitivity indicates that LLMs are substantially more susceptible to bias based on their prior knowledge (as compared to humans). These findings raise crucial questions and have broad implications for human-AI and AI-AI communication of uncertainty.