From Unstructured Communication to Intelligent RAG: Multi-Agent Automation for Supply Chain Knowledge Bases

Yao Zhang, Zaixi Shang, Silpan Patel, Mikel Zuniga

Keywords: Generative AI, LLM, Retrieval-Augmented Generation (RAG), Multi-Agent System, Offline Knowledge Base Construction, Category Discovery, Categorization, Summarization

TL;DR: We present a multi-agent system that transforms unstructured supply chain support communications into structured knowledge bases offline, compressing ticket volume by 96.6% while improving RAG-based automated resolution rates to ~50%.

Abstract: Supply chain operations generate vast amounts of operational data; however, critical knowledge—such as system usage practices, troubleshooting workflows, and resolution techniques—often remains buried within unstructured communications like support tickets, emails, and chat logs. While Retrieval-Augmented Generation (RAG) systems aim to leverage such communications as a knowledge base, their effectiveness is limited by raw data challenges: support tickets and chat logs are typically noisy, inconsistent, and incomplete, making direct retrieval suboptimal. Unlike existing RAG approaches that focus on runtime optimization, we introduce a novel offline-first methodology that transforms these communications into a structured knowledge base. Our key innovation is a Large language models (LLMs)-based multi-agent system orchestrating three specialized agents: Category Discovery for taxonomy creation, Categorization for ticket grouping, and Knowledge Synthesis for article generation. Applying our methodology to real-world support tickets with resolution notes and comments between resolvers and requesters, our system creates a compact knowledge base—reducing the total volume to just 3.4\% of the original ticket data while improving quality. Experiments demonstrate that plugging our prebuilt knowledge base into a RAG system significantly outperforms traditional RAG implementations (48.74\% vs. 38.60\% helpful answers) and achieves a 77.4\% reduction in unhelpful responses. By automating institutional knowledge capture that typically remains siloed in experts' heads, our solution translates to substantial operational efficiency: reducing support workload, accelerating resolution times, and creating self-improving systems that can automatically resolve approximately 50\% of future supply chain tickets. Our approach addresses a key gap in knowledge management by transforming transient communications into structured and reusable knowledge base through intelligent offline processing rather than latency-inducing runtime architectures.

Submission Number: 5