RAG Knowledge Online Correction with Conversation-Based User Feedback

Yifan Liu, Changjian Chen, Zhuo Tang, Haixin Xie, Kenli Li

Keywords: Retrieval-Augmented Generation, Multi-Agent Systems, Dynamic Action Search

Abstract: Retrieval-Augmented Generation (RAG) is a promising way to enhance LLMs by integrating external knowledge. However, its performance degrades when the knowledge contains errors. When users encounter such errors, the typical correction process involves users reporting the errors, after which server providers investigate the knowledge base to identify and fix the issues. This process is often time-consuming, and in the meantime, other users continue to encounter the same errors, leading to poor user experience. To address this challenge, we propose a new task, Knowledge Online Correction, which focuses on correcting errors immediately after they are pointed out by users through conversation-based feedback. To evaluate this task, we conducted a preliminary user study and developed a new benchmark, ConvCorrect. To address this task, we propose a Multi-step Knowledge Online Correction method MT-KOC, an online knowledge correction method that automatically corrects errors in real time based on a dynamic action search algorithm. Empirical results shows that MT-KOC outperforms baseline methods, achieving higher accuracy in the knowledge online correction task.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 2574