KPI-Chain: Multi-Agent Planning with Entity-Based Task Chaining for Reliable Recovery

Yossi Ben Shlomo, Dror Cohen, Eviatar Hutah, Neri richter, Orel Balilti

Keywords: multi-agent systems, large language models, automated planning, failure recovery, entity extraction, key performance indicators, task chaining, JSON-path memory, Model Context Protocol, agent reliability

Abstract: Planning-based LLM agent frameworks promise flexible problem-solving through structured task decomposition, but they remain brittle: plans often fail silently, and existing approaches lack mechanisms for reliable recovery. We propose KPI-Chain, a multi-agent planning framework with a novel plan structure that embeds per-task key performance indicators (KPIs) based on typed entity extraction. This plan design—our core contribution—fundamentally improves agent reliability by making task specifications more precise and explicit upfront. In our formulation, each task explicitly defines expected entities (string, number, array, dict) to be extracted from its output. This structure drives multiple benefits throughout the system: it forces clearer, more specific task definitions during planning; it focuses extraction on only the relevant key information from tool responses rather than verbose outputs; it enables reasoning tasks to produce structured, targeted results; it supports efficient and precise memory management through typed entity storage; and critically, it makes failure root causes immediately identifiable when expected entities cannot be extracted. When KPIs are not met, the system automatically triggers continuation-based replanning with explicit failure feedback. To operationalize this plan structure, we introduce complementary components: an entity extractor for validating KPIs, a JSON-path memory system for typed entity storage and retrieval, MCP integration for standardized tool access, and chain-of-thought prompting for reasoning tasks. Across 5 challenging benchmarks, our KPI-Chain framework achieves higher success rates compared to existing agent architectures including ReAct and Plan-and-Execute. These results demonstrate that embedding entity-based KPIs directly into plan structure provides a foundation for building more reliable and adaptive LLM agent systems.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 13987