Evaluating the Quality of AI-Generated Resolutions from Conversational vs Structured Sources: Implications for Enterprise Knowledge Automation

Archan Dutta, Hardik Airen, Phani Nivarthi, Vinay Sisodiya

Keywords: AI-Driven Automation, Knowledge Mining, Resolution Extraction from Conversational Sources, Quality Evaluation using LLMs

TL;DR: We present a pipeline that extracts high-quality resolutions from noisy conversational data (Slack) and compare them against structured ticket-based resolutions, showing statistical differences across key reliability metrics.

Abstract:

Enterprises increasingly rely on historical data to extract resolutions for problems and to automate knowledge mining.
While structured ticketing systems (such as ServiceNow, Freshservice, Zendesk etc.) are well-established sources for resolutions, conversational platforms like Slack also capture valuable knowledge in less formal contexts. This paper proposes a Resolution Extraction System to extract meaningful resolutions for IT support cases from noisy, unstructured conversational platforms like Slack, MS Teams, etc. The paper then compares these AI-generated resolutions extracted from Slack conversations (RES) to AI-generated resolutions extracted from structured ticketing systems (RET). We evaluate six key performance indicators (KPIs) - context relevance, completeness, conciseness, noise, perplexity, and readability across 1,000 samples.
Our results reveal systematic differences between structured and conversational sources. The analysis shows that with high-precision filtering, conversational sources can be transformed into a meaningful source of resolutions despite the challenges of building reliable enterprise knowledge systems from noisy data. Slack-based resolutions are more relevant and concise but noisier and less readable, whereas ticket-based resolutions are more structured and easier to interpret. These findings highlight the complementary role of conversational data for enterprise knowledge mining and provide guidance on integrating multiple sources into AI-driven automation for support and resolution.

Submission Number: 169