PermitQA: A Benchmark for Retrieval Augmented Generation in Wind Siting and Permitting domain
Abstract: In the rapidly evolving landscape of Natural Language Processing (NLP) and text generation, the emergence of Retrieval Augmented Generation (RAG) presents a promising avenue for improving the quality and reliability of generated text by leveraging information retrieved from user specified database. Benchmarking is essential to evaluate and compare the performance of the different RAG configurations in terms of \emph{retriever} and \emph{generator}, providing insights into their effectiveness, scalability, and suitability for the specific domain and applications. In this paper, we present a comprehensive framework to generate a domain relevant RAG benchmark. Our framework is based on automatic question-answer generation with Human (domain experts)-AI Large Language Model (LLM) teaming. As a case study, we demonstrate the framework by introducing PermitQA, a first-of-its-kind benchmark on the wind siting and permitting domain which comprises of multiple scientific documents/reports related to environmental impact of wind energy projects. Our framework systematically evaluates RAG performance using diverse metrics and multiple question types with varying complexity level. We also demonstrate the performance of different models on our benchmark.
Paper Type: Long
Research Area: Language Modeling
Research Area Keywords: benchmarking, evaluation methodologies, evaluation, metrics, prompting
Contribution Types: Model analysis & interpretability, Data analysis
Languages Studied: English
Submission Number: 293
Loading