Theory of LLM sampling: part descriptive and part prescriptive

Sarath Sivaprasad, Pramod Kaushik, Sahar Abdelnabi, Mario Fritz

Keywords: decision making, sampling, prescriptive norm, heuristics

TL;DR: We investigate how Large Language Models (LLMs) sample responses and propose that their outputs are influenced by both descriptive (statistical averages) and prescriptive (ideal) components.

Abstract:

Large Language Models (LLMs) are increasingly utilized in autonomous decision-making systems, where they sample options from an action space. However, the underlying heuristics guiding the sampling of LLMs remain under-explored. We examine LLM response sampling and propose a theory that the sample of an LLM is driven by a descriptive component (the notion of statistical average) and a prescriptive component (notion of an ideal represented in the LLM). In a controlled experimental setting, we demonstrate that LLM outputs deviate from statistically probable outcome in the direction of a presciptive component. We further show this deviation towards prescriptive component consistently appears across diverse real-world domains, including social, public health, and scientific contexts. Using this theory, we show that concept prototypes in LLMs are affected by prescriptive norms, similar to concept of normality in humans. Through case studies, we illustrate that in real-world applications, the shift toward an ideal value in LLM outputs can result in significantly biased decision-making, raising ethical concerns.

Primary Area: applications to neuroscience & cognitive science

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8159