SocraticAgent: An Autonomous Agent for Unlocking Latent Knowledge in LLMs

Yang Yan, Yu Lu, Renjun Xu, Zhenzhong Lan

Keywords: Autonomous Agents, Agent Architectures, Large Language Models, Knowledge Representation, Automated Reasoning

TL;DR: We introduce SocraticAgent, a method that enhances an LLM's ability to recall its own internal knowledge, enabling accurate and self-reliant question answering without the need for external retrieval systems.

Abstract: Reasoning failures in Large Language Models (LLMs) used by autonomous agents are often attributed to knowledge deficits, leading to a reliance on solutions like Retrieval-Augmented Generation (RAG) or parametric fine-tuning. This paper empirically demonstrates that this assumption is often flawed. We identify a quantifiable "knowledge recall gap": while modern LLMs possess 90-97\% of the necessary facts for a task, they spontaneously apply only 57-64\% of this knowledge during reasoning. This reveals a significant performance gap rooted in a failure of recall, not a fundamental absence of knowledge. To address this, we introduce SocraticAgent, a zero-shot autonomous agent that emulates Socratic inquiry by guiding an LLM to first deconstruct a problem and comprehensively detail the internal knowledge required for its solution. Through a deterministic two-action cycle of (1) knowledge deconstruction and (2) grounded reasoning, it procedurally closes this recall gap without any model updates. Across a diverse suite of LLMs, SocraticAgent significantly improves reasoning accuracy, outperforming standard prompting and noisy external retrieval. Critically, our agentic, process-driven approach achieves performance competitive with expensive, data-dependent fine-tuning methods, but does so at inference time without any parametric changes. Our work provides the first clear demonstration that a deliberative agentic process can serve as a powerful and highly efficient substitute for parametric memory adaptation. This paves the way for lighter, more adaptable, and more capable autonomous reasoning systems, positioning agent-driven deliberation as a key mechanism for unlocking the vast latent knowledge within LLMs.

Area: Generative and Agentic AI (GAAI)

Generative A I: I acknowledge that I have read and will follow this policy.

Submission Number: 1053