Improving LLM Agent Planning with In-Context Learning via Atomic Fact Augmentation and Lookahead Search

Samuel Holt, Max Ruiz Luyten, Thomas Pouplin, Mihaela van der Schaar

Keywords: LLM Agents, Programmatic Representations, In-Context Learning, Learned Programmatic Facts, World Modeling, Planning, Lookahead Search, Sequential Decision-Making, Interpretability

TL;DR: An LLM agent learns programmatic textual facts as structured in-context knowledge to enhance its planning and internal world model via lookahead search.

Track: Long Paper (up to 9 pages)

Abstract: Large Language Models (LLMs) are increasingly capable but often require significant guidance or extensive interaction history to perform effectively in complex, interactive environments. Existing methods may struggle with adapting to new information or efficiently utilizing past experiences for multi-step reasoning without fine-tuning. We introduce a novel LLM agent framework that enhances planning capabilities through in-context learning, facilitated by atomic fact augmentation and a recursive lookahead search. Our agent learns to extract task-critical ``atomic facts'' from its interaction trajectories. These facts dynamically augment the prompts provided to LLM-based components responsible for action proposal, latent world model simulation, and state-value estimation. Planning is performed via a depth-limited lookahead search, where the LLM simulates potential trajectories and evaluates their outcomes, guided by the accumulated facts and interaction history. This approach allows the agent to improve its understanding and decision-making online, leveraging its experience to refine its behavior without weight updates. We provide a theoretical motivation linking performance to the quality of fact-based abstraction and LLM simulation accuracy. Empirically, our agent demonstrates improved performance and adaptability on challenging interactive tasks, achieving more optimal behavior as it accumulates experience, showcased in tasks such as TextFrozenLake and ALFWorld.

Format: We have read the camera-ready instructions, and our paper is formatted with the provided template.

De-Anonymization: This submission has been de-anonymized.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Submission Number: 31