Perceptions to Beliefs: Exploring Precursory Inferences for Theory of Mind in Large Language Models
Abstract: While humans naturally develop theory of mind (ToM), the capability to understand other people's mental states and beliefs, state-of-the-art large language models (LLMs) underperform on simple ToM benchmarks.
We posit that we can extend our understanding about LLMs' ToM abilities by evaluating key human ToM precursors--perception inference and perception-to-belief inference--in LLMs.
We introduce two datasets, Percept-ToMi and Percept-FANToM, to evaluate these precursory inferences for ToM in LLMs by annotating characters' perceptions within two existing ToM benchmarks, ToMi and FANToM.
Our evaluation of eight state-of-the-art LLMs reveals that the models perform generally well in perception inference while exhibiting limited capability in perception-to-belief inference.
Based on these results, we present PercepToM, a novel ToM method leveraging LLMs' strong perception inference capability while supplementing their limited perception-to-belief inference.
Experimental results demonstrate that PercepToM significantly enhances LLM performance on the ToMi and FANToM benchmarks, especially in false belief scenarios.
Paper Type: Long
Research Area: Dialogue and Interactive Systems
Research Area Keywords: theory of mind, evaluation, large language models, social reasoning
Contribution Types: Model analysis & interpretability, NLP engineering experiment, Data resources
Languages Studied: English
Submission Number: 1240
Loading