Mindless Dialogue? A Critical Note on ToM and Communicative Generative Agents

Effat Farhana

Keywords: Theory of Mind, Generative AI, Communicating agents

Abstract: Theory of Mind (ToM), the ability to attribute beliefs, desires, and intentions to others, plays a foundational role in human communication, language acquisition, and social interaction. The rise of generative models capable of open-ended dialogue raises important questions about whether these systems truly use ToM-like reasoning or simply mimic the surface-level behaviors associated with it, such as predicting others’ thoughts or intentions. While recent work suggests that large language models (LLMs) exhibit some competencies associated with ToM, such abilities often emerge from statistical pattern recognition rather than grounded reasoning about mental states. In this commentary, we critically examine the assumption that generative agents "understand" or "reason" about others' minds during communication. We highlight the distinction between genuine ToM capacities and the appearance of ToM arising from training on massive datasets. Drawing on insights from both cognitive science and machine learning, we argue that many current benchmarks and evaluations fail to capture the nuance of ToM as a developmental and socially grounded process. We also examine the risks of attributing human-like understanding to generative agents, particularly in socially sensitive settings where the illusion of comprehension may lead to overtrust or potential manipulation. In alignment with the ACS focus, this commentary urges the community to rethink how ToM is conceptualized, implemented, and evaluated in communicative AI. We advocate for interdisciplinary approaches that go beyond behavioral proxies, emphasize developmental insights, and consider the broader social implications of deploying agents that appear to "know what we mean" even when they do not.

Paper Track: Commentary

Submission Number: 40