Uncertainty Beneath the Surface: A Study of How Language Models Enocde Linguistic Uncertainty

ACL ARR 2025 July Submission1050 Authors

Abstract: Large language models (LLMs) are increasingly being deployed in high-stakes domains where the ability to reason under uncertainty is critical. Despite recent progress in evaluating factuality and calibration, little is known about how LLMs internally represent epistemic modality: linguistic cues that signal speaker uncertainty (e.g., ``might'', ``probably''). This work presents one of the first systematic investigations into whether and how LLMs encode sensitivity to epistemic modality in their activation space. We curate a contrastive multiple choice dataset of \textit{3114} sentence pairs that vary in epistemic certainty and introduce a probing framework to quantify activation-level differences between certain and uncertain prompts. We further propose Model Sensitivity to Uncertainty (MSU), a layerwise metric that captures representational shifts attributable to epistemic cues. Our findings suggest that LLMs exhibit measurable and layer-specific sensitivity to epistemic modality, raising implications for their deployment in sensitive decision-making contexts.

Paper Type: Short

Research Area: Interpretability and Analysis of Models for NLP

Research Area Keywords: Interpretability and Analysis of Models for NLP

Contribution Types: Model analysis & interpretability, Reproduction study, Publicly available software and/or pre-trained models, Data resources, Surveys

Languages Studied: English

Reassignment Request Area Chair: This is not a resubmission

Reassignment Request Reviewers: This is not a resubmission

A1 Limitations Section: This paper has a limitations section.

A2 Potential Risks: N/A

B Use Or Create Scientific Artifacts: Yes

B1 Cite Creators Of Artifacts: Yes

B1 Elaboration: 2

B2 Discuss The License For Artifacts: Yes

B2 Elaboration: Appendix

B3 Artifact Use Consistent With Intended Use: Yes

B3 Elaboration: 2

B4 Data Contains Personally Identifying Info Or Offensive Content: No

B4 Elaboration: The data that was collected/used does not contain any information that names or uniquely identifies individual people or offensive content.

B5 Documentation Of Artifacts: Yes

B5 Elaboration: Appendix

B6 Statistics For Data: Yes

B6 Elaboration: 2

C Computational Experiments: Yes

C1 Model Size And Budget: Yes

C1 Elaboration: Appendix

C2 Experimental Setup And Hyperparameters: Yes

C2 Elaboration: Appendix

C3 Descriptive Statistics: Yes

C3 Elaboration: Appendix

C4 Parameters For Packages: Yes

C4 Elaboration: 2

D Human Subjects Including Annotators: No

D1 Instructions Given To Participants: N/A

D2 Recruitment And Payment: N/A

D3 Data Consent: N/A

D4 Ethics Review Board Approval: N/A

D5 Characteristics Of Annotators: N/A

E Ai Assistants In Research Or Writing: Yes

E1 Information About Use Of Ai Assistants: No

E1 Elaboration: Assistance with writing only.

Author Submission Checklist: yes

Submission Number: 1050