LLMs Can't Play Hangman: On the Necessity of a Private Working Memory for Language Agents

ACL ARR 2026 January Submission2324 Authors

Keywords: Large Language Models, AI Agents, Memory Agents, Private Working Memory, Private State Interactive Tasks (PSITs), State Persistence, Self-Consistency Test, Cognitive Architectures, Memory-Augmented LLMs

Abstract:

As LLMs move from text completion toward autonomous agents, they remain constrained by the standard chat interface, which lacks private working memory. This raises a fundamental question: can agents reliably perform interactive tasks that depend on hidden state?
We define Private State Interactive Tasks (PSITs), which require agents to generate and maintain hidden information while producing consistent public responses. We show theoretically that any agent restricted to the public conversation history cannot simultaneously preserve secrecy and consistency in PSITs, yielding an impossibility theorem.
To empirically validate this limitation, we introduce a self-consistency testing protocol that evaluates whether agents can maintain a hidden secret across forked dialogue branches. Standard chat-based LLMs and retrieval-based memory baselines fail this test regardless of scale, demonstrating that semantic retrieval does not enable true state maintenance. To address this, we propose a novel architecture incorporating an explicit private working memory; we demonstrate that this mechanism restores consistency, establishing private state as a necessary component for interactive language agents. Our code is available at https://anonymous.4open.science/r/Hangman-6B6F.

Paper Type: Long

Research Area: AI/LLM Agents

Research Area Keywords: LLM/AI agents, chain-of-thought, prompting, safety and alignment, evaluation and metrics, explanation faithfulness

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Publicly available software and/or pre-trained models, Data resources, Theory

Languages Studied: English

Submission Number: 2324