OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain

ACL ARR 2025 February Submission4043 Authors

Abstract:

Retrieval-augmented generation (RAG) has emerged as a key application of large language models (LLMs), especially in vertical domains where LLMs may lack domain-specific knowledge. This paper introduces OmniEval, an omnidirectional and automatic RAG benchmark for the financial domain, featured by its multi-dimensional evaluation framework:
First, we categorize RAG scenarios by five task classes and 16 financial topics, leading to a matrix-based structured assessment for RAG evaluation;
Next, we leverage a multi-dimensional evaluation data generation method that integrates GPT-4-based automatic generation and human annotation approaches, achieving an 87.47\% acceptance ratio in human evaluations of generated instances;
Further, we utilize a multi-stage evaluation pipeline to assess both retrieval and generation performance, resulting in an all-sided evaluation of the RAG pipeline.
Finally, rule-based and LLM-based metrics are combined to build a multi-dimensional evaluation system, enhancing the reliability of assessments through fine-tuned LLM-based evaluators.
Our omnidirectional evaluation experiments highlight the performance variations of RAG systems across diverse topics and tasks and reveal significant opportunities for RAG models to improve their capabilities in vertical domains.
We open source the anonymous code of our benchmark at https://anonymous.4open.science/r/OmniEval-anonymous-8E48

Paper Type: Long

Research Area: Resources and Evaluation

Research Area Keywords: retrieval-augmented generation,automatic evaluation,financial/business NLP

Contribution Types: Data resources

Languages Studied: English,Chinese

Submission Number: 4043