Unanswerability Evaluation for Retreival Augmented Generation

ACL ARR 2024 December Submission1460 Authors

Abstract:

Existing evaluation frameworks for retrieval-augmented generation (RAG) systems focus on answerable queries, but they overlook the importance of appropriately rejecting unanswerable requests.
In this paper, we introduce UAEval4RAG, a framework designed to evaluate whether RAG systems can handle unanswerable queries effectively.
We define a taxonomy with six unanswerable categories, and UAEval4RAG automatically synthesizes diverse and challenging queries for any given knowledge base with unanswered ratio and acceptable ratio metrics.
We conduct experiments with various RAG components, including retrieval models, rewriting methods, rerankers, language models, and prompting strategies, and reveal hidden trade-offs in performance of RAG systems.
Our findings highlight the critical role of component selection and prompt design in optimizing RAG systems to balance the accuracy of answerable queries with high rejection rates of unanswerable ones.
UAEval4RAG provides valuable insights and tools for developing more robust and reliable RAG systems.

Paper Type: Long

Research Area: Resources and Evaluation

Research Area Keywords: Retreival Augmented Generation, evaluation, large language models

Contribution Types: Publicly available software and/or pre-trained models

Languages Studied: English

Submission Number: 1460