Eliminating Retrieval Knowledge Conflicts: Cross-Validated Re-ranking with Large Language Models
Abstract: In retrieval-augmented generation systems, employing large language models for re-ranking has proven effective. However, existing work often prioritizes passage relevance over reliability, leading to the utilization of conflicting information and the generation of ambiguous answers. This is particularly problematic when dealing with inter-context knowledge conflicts, where candidate documents contain opposing information that can mislead the model. To address this issue, we introduce a novel cross-validation re-ranking technique that specifically resolves these inter-context knowledge conflicts during retrieval. We develope a new dataset, ContraPRT, specifically to test the model's ability to rank sets of passages containing conflicting knowledge. Results with GPT-4 and LlaMA3-70B demonstrate that our approach not only successfully filters out conflicting information but also ensures that the passage rankings are accurate, thus providing reliable supplementary knowledge for the generation module.
Paper Type: Long
Research Area: Information Retrieval and Text Mining
Research Area Keywords: Passage Re-ranking, Knowledge Conflicts, Large Language Models
Contribution Types: Model analysis & interpretability, NLP engineering experiment
Languages Studied: English
Submission Number: 4088
Loading