RAG Knowledge Online Correction with Conversation-Based User Feedback

ACL ARR 2026 January Submission1759 Authors

Keywords: Retrieval-Augmented Generation; Knowledge Correction; Multi-agent Systems; Benchmark

Abstract: Retrieval-Augmented Generation (RAG) is a promising paradigm to enhance LLMs by integrating external knowledge. However, its performance degrades when the knowledge contains errors. When users encounter such errors, the typical correction process involves users reporting the errors, after which service providers investigate the knowledge base to identify and fix the issues. This process is often time-consuming, and in the meantime, other users continue to encounter the same errors, leading to a poor user experience. To address this challenge, we propose a new task, Knowledge Online Correction, which focuses on correcting errors immediately after they are pointed out by users through conversation-based feedback. To evaluate this task, we conducted a preliminary user study and, based on the findings, developed a new benchmark, ConvCorrect. To address this task, we propose MT-KOC (Multi-step Knowledge Online Correction), a multi-agent framework that utilizes a dynamic action search algorithm to identify the optimal correction sequence for progressive correction. Empirical results on ConvCorrect show that our method significantly outperforms existing baselines. Our code is available at https://anonymous.4open.science/r/MT-KOC.

Paper Type: Long

Research Area: Retrieval-Augmented Language Models

Research Area Keywords: retrieval-augmented generation; knowledge base QA; LLM/AI agents

Contribution Types: NLP engineering experiment, Data resources

Languages Studied: English

Submission Number: 1759