From unstructured communication to intelligent RAG: Multi-agent automation for supply chain knowledge bases

Supply chain operations generate vast amounts of operational data; however, critical knowledge—such as system usage practices, troubleshooting workflows, and resolution techniques—often remains buried within unstructured communications like support tickets, emails, and chat logs. While Retrieval-Augmented Generation (RAG) systems aim to leverage such communications as a knowledge base, their effectiveness is limited by raw data challenges: support tickets and chat logs are typically noisy, inconsistent, and incomplete, making direct retrieval suboptimal. Unlike existing RAG approaches that focus on runtime optimization, we introduce a novel offline-first methodology that transforms these communications into a structured knowledge base. Our key innovation is a Large language models (LLMs)-based multi-agent system orchestrating three specialized agents: Category Discovery for taxonomy creation, Categorization for ticket grouping, and Knowledge Synthesis for article generation. Applying our methodology to real-world support tickets with resolution notes and comments between resolvers and requesters, our system creates a compact knowledge base—reducing the total volume to just 3.4% of the original ticket data while improving quality. Experiments demonstrate that plugging our prebuilt knowledge base into a RAG system significantly outperforms traditional RAG implementations (48.74% vs. 38.60% helpful answers) and achieves a 77.4% reduction in unhelpful responses. By automating institutional knowledge capture that typically remains siloed in experts’ heads, our solution translates to substantial operational efficiency: reducing support workload, accelerating resolution times, and creating self-improving systems that can automatically resolve approximately 50% of future supply chain tickets. Our approach addresses a key gap in knowledge management by transforming transient communications into structured and reusable knowledge base through intelligent offline processing rather than latency-inducing runtime architectures.