RPO: Retrieval Preference Optimization for Robust Retrieval-Augmented Generation

ACL ARR 2025 February Submission3026 Authors

Abstract: While Retrieval-Augmented Generation (RAG) has exhibited promise in utilizing external knowledge, its generation process heavily depends on the quality and accuracy of the retrieved context. Large language models (LLMs) struggle to evaluate the correctness of non-parametric knowledge retrieved externally when it differs from internal memorization, leading to *knowledge conflicts* during response generation. To this end, we introduce the **R**etrieval **P**reference **O**ptimization (RPO), a lightweight and effective alignment method to adaptively leverage multi-source knowledge based on retrieval relevance. An implicit representation of retrieval relevance is derived and incorporated into the reward model to integrate retrieval evaluation and response generation into a single model, solving the problem that previous methods necessitate the additional procedure to assess the retrieval quality. Notably, RPO is a RAG-dedicated alignment approach that quantifies the awareness of retrieval relevance in training, first overcoming mathematical obstacles. Experiments on four datasets demonstrate that RPO outperforms RAG by 4-10% in accuracy without any extra component, exhibiting its robust generalization.

Paper Type: Long

Research Area: Generation

Research Area Keywords: Retrieval-Augmented Generation, Model Aligment, Retrieval Preference Optimization, Robust Generation

Contribution Types: Approaches low compute settings-efficiency

Languages Studied: English

Submission Number: 3026