RAG-Reward: Optimizing RAG with Reward Modeling and RLHF

ACL ARR 2025 February Submission5122 Authors

Abstract: Retrieval-augmented generation (RAG) enhances Large Language Models (LLMs) with relevant and up-to-date knowledge, improving their ability to answer knowledge-intensive questions. It has been shown to enhance both generation quality and trustworthiness. While numerous works have focused on improving retrieval, generation, and evaluation, the role of reward models in reinforcement learning for optimizing RAG remains underexplored. In this paper, we introduce RAG-Reward, a framework designed to develop reward models to enable hallucination-free, comprehensive, reliable, and efficient RAG. We define four key metrics to assess generation quality and develop an automated benchmarking pipeline to evaluate the outputs of multiple LLMs across a variety of RAG scenarios. Using RAG-Reward, we train reward models and apply reinforcement learning with human feedback (RLHF) to improve LLMs' effectiveness in RAG. Experimental results demonstrate that our reward model achieves state-of-the-art performance in automatic benchmarking and aligns closely with human evaluations. Furthermore, the improved generation quality of the trained policy model highlights the feasibility and efficiency of using RLHF to enhance RAG outputs.

Paper Type: Long

Research Area: NLP Applications

Research Area Keywords: Reward Modeling, Retrieval-Augmented Generation, RLHF

Contribution Types: NLP engineering experiment

Languages Studied: English

Submission Number: 5122