Truth-value judgment in language models: belief directions are context sensitive

Stefan F. Schouten, Peter Bloem, Ilia Markov, Piek Vossen

Keywords: interpretability, truth directions, LLM beliefs, large language model, llm

Abstract:

Recent work has demonstrated that the latent spaces of large language models
(LLMs) contain directions predictive of the truth of sentences. Multiple methods recover such directions and build probes that are described as uncovering
a model’s “knowledge” or “beliefs”. We investigate this phenomenon, looking
closely at the impact of context on the probes. Our experiments establish where
in the LLM the probe’s predictions are (most) sensitive to the presence of related
sentences, and how to best characterize this kind of sensitivity. We do so by
measuring different types of consistency errors that occur after probing an LLM
whose inputs consist of hypotheses preceded by (negated) supporting and contradicting sentences. We also perform a causal intervention experiment, investigating whether moving the representation of a premise along these belief directions
influences the position of an entailed or contradicted sentence along that same direction. We find that the probes we test are generally context sensitive, but that
contexts which should not affect the truth often still impact the probe outputs.
Our experiments show that the type of errors depend on the layer, the model, and
the kind of data. Finally, our results suggest that belief directions are (one of the)
causal mediators in the inference process that incorporates in-context information.

Primary Area: interpretability and explainable AI

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 10399