Triggering RAG through LLM Judgments on Expressed Unknown Knowledge

ACL ARR 2024 December Submission1814 Authors

Abstract: Large Language Models (LLMs) often struggle with dynamically changing knowledge and handling unknown static information. Retrieval-Augmented Generation (RAG) is employed to tackle these challenges and has a significant impact on improving LLM performance. In fact, we find that not all questions need to trigger RAG. By retrieving parts of knowledge unknown to the LLM and allowing the LLM to answer the rest, we can effectively reduce both time and computational costs. In our work, we propose a Knowledge Boundary Model (KBM) to express the known/unknown of a given question, and to determine whether a RAG needs to be triggered. Experiments conducted on 11 English and Chinese datasets illustrate that the KBM effectively delineates the knowledge boundary, significantly decreasing the proportion of retrievals required for optimal end-to-end performance. Furthermore, we evaluate the effectiveness of KBM in three complex scenarios: dynamic knowledge, long-tail static knowledge, and multi-hop problems, as well as its functionality as an external LLM plug-in.

Paper Type: Long

Research Area: Question Answering

Research Area Keywords: Large Language Models, Question Answering, Retrieval-Augmented Generation

Contribution Types: NLP engineering experiment, Approaches to low-resource settings

Languages Studied: English, Chinese

Submission Number: 1814