CausalPlan: Empowering Efficient LLM Multi-Agent Collaboration Through Causality-Driven Planning

ICLR 2026 Conference Submission16296 Authors

Keywords: Reinforcement Learning, Large Language Models, Causality, Collaborative Planning, Embodied Agent

TL;DR: We propose CausalPlan, a framework that embeds structural causal model into LLM planning. Using a learned causal graph to score and reweight LLM-generated plans, CausalPlan reduces invalid actions and improves coordination.

Abstract: Large language model (LLM) agents often generate causally invalid plans in collaborative tasks due to their reliance on surface-level correlations rather than grounded causal reasoning. This limitation undermines their performance in terms of coordination and planning in dynamic environments. We address this challenge with CausalPlan, a framework that integrates explicit structural causal reasoning into the LLM planning process. At the core of CausalPlan is the Structural Causal Action (SCA) model, which learns a causal graph from agent trajectories to capture how prior actions and current environment states influence future decisions. This model is then used to inform the planning process, shaping proposed LLM-generated plans through causal scoring, reweighting, and fallback to grounded alternatives when needed. By embedding this causal knowledge directly into the decision loop, CausalPlan constrains planning to intervention-consistent behaviors without requiring fine-tuning. We evaluated CausalPlan on the Overcooked-AI benchmark across five multi-agent coordination tasks and four LLMs of varying sizes: Gemma-7B, Llama-8B, Qwen-14B and Llama-70B. Experimental results show that CausalPlan consistently reduces invalid actions and improves collaboration in both AI-AI and human-AI settings, outperforming strong reinforcement learning baselines. Our findings highlight the value of causality-driven planning for deploying efficient, interpretable, and generalisable multi-agent LLM systems.

Primary Area: applications to robotics, autonomy, planning

Submission Number: 16296