Adaptive Retrieval-Augmented Generation for Conversational Systems
Abstract: Despite the success of integrating large language models into the development of conversational systems, many studies have shown the effectiveness of retrieving and augmenting external knowledge for informative responses. Hence, many existing studies commonly assume the *always* need for Retrieval Augmented Generation (RAG) in a conversational system without explicit control. This raises a research question about such a necessity. In this study, we propose to investigate the need for each turn of system response to be augmented with external knowledge. In particular, by leveraging human judgements on the binary choice of adaptive augmentation, we develop *RAGate*, a gating model, which models conversation context and relevant inputs to predict if a conversational system requires RAG for improved responses. We conduct extensive experiments on devising and applying *RAGate* to conversational models and well-rounded analyses of different conversational scenarios. Our experimental results and analysis indicate the effective application of *RAGate* in RAG-based conversational systems in identifying system responses for appropriate RAG with high-quality responses in a high generation confidence. This study also identifies the correlation between the generation's confidence level and the relevance of the augmented knowledge.
Paper Type: Long
Research Area: Dialogue and Interactive Systems
Research Area Keywords: retrieve-augmented generation, conversational system, task-oriented dialogue system
Contribution Types: Model analysis & interpretability, NLP engineering experiment
Languages Studied: English
Submission Number: 2376
Loading