Unanswerability Evaluation for Retrieval Augmented Generation

ACL ARR 2025 February Submission5311 Authors

Abstract:

Existing evaluation frameworks for retrieval-augmented generation (RAG) systems focus on answerable queries, but they overlook the importance of appropriately rejecting unanswerable requests. In this paper, we introduce UAEval4RAG, a comprehensive evaluation framework designed to evaluate whether RAG systems effectively handle unanswerable queries specific to a given knowledge base.
We first define a taxonomy with six unanswerable categories, and UAEval4RAG automatically synthesizes diverse and challenging queries for any given knowledge base and evaluate the RAG systems with unanswered ratio and acceptable ratio metrics.
We also conduct experiments with various RAG components and prompting strategies across four datasets,
which reveals that due to varying knowledge distribution across datasets, no single configuration consistently delivers optimal performance on both answerable and unanswerable requests across different knowledge bases.
Our findings highlight the critical role of component selection and prompt design in optimizing RAG systems to balance the accuracy of answerable queries with high rejection rates of unanswerable ones. UAEval4RAG provides valuable insights and tools for developing more robust and reliable RAG systems.

Paper Type: Long

Research Area: Resources and Evaluation

Research Area Keywords: Retreival Augmented Generation, evaluation, large language models

Contribution Types: Publicly available software and/or pre-trained models

Languages Studied: English

Submission Number: 5311