Understanding the Relationship between Prompts and Response Uncertainty in Large Language Models
Abstract: Large language models (LLMs) are widely used in decision-making, but their reliability, especially in critical tasks like healthcare, is not well-established. Therefore, understanding how LLMs reason and make decisions is crucial for their safe deployment. This paper investigates how the uncertainty of responses generated by LLMs relates to the information provided in the input prompt. Leveraging the insight that LLMs learn to infer latent concepts during pretraining, we propose a prompt-response concept model that explains how LLMs generate responses and helps understand the relationship between prompts and response uncertainty. We show that the uncertainty decreases as the prompt's informativeness increases, similar to epistemic uncertainty. Our detailed experimental results on real datasets validate our proposed model.
Paper Type: Long
Research Area: Interpretability and Analysis of Models for NLP
Research Area Keywords: Uncertainty quantification
Contribution Types: Model analysis & interpretability, Theory
Languages Studied: English
Submission Number: 5825
Loading