PRP-Graph: Pairwise Ranking Prompting to LLMs with Graph Aggregation for Effective Text Re-ranking

Anonymous

Abstract: Pairwise Ranking Prompting (PRP) demonstrates impressive effectiveness in zero-shot document re-ranking tasks with large language models (LLMs). However, in the existing methods, PRP only outputs the same label for the comparison results of different confidence intervals without considering the uncertainty of pairwise comparison, which implies an underutilization of the generation probability information of LLMs. To bridge this gap, we propose PRP-Graph, a novel pairwise re-ranking approach, based on a refined scoring PRP unit that exploits the output probabilities of target labels to capture the degree of certainty of the comparison results. Specifically, the PRP-Graph consists of two stages, namely ranking graph construction and ranking graph aggregation. Extensive experiments conducted on the BEIR benchmark demonstrate the superiority of our approach over existing PRP-based methods. Comprehensive analysis reveals that the PRP-Graph displays strong robustness towards the initial ranking order and delivers exceptional re-ranking results with acceptable efficiency. Our code and data will be available.

Paper Type: long

Research Area: Information Retrieval and Text Mining

Contribution Types: NLP engineering experiment, Theory

Languages Studied: English

Preprint Status: We are considering releasing a non-anonymous preprint in the next two months (i.e., during the reviewing process).

A1: yes

A1 Elaboration For Yes Or No: Section Limitations on page 9.

A2: no

A2 Elaboration For Yes Or No: There is no significant potential risk in our work.

A3: yes

A3 Elaboration For Yes Or No: 1

B: yes

B1: yes

B1 Elaboration For Yes Or No: 4.1

B2: no

B2 Elaboration For Yes Or No: We follow the licenses disclosed by the artifacts we use.

B3: yes

B3 Elaboration For Yes Or No: 4.1

B4: no

B4 Elaboration For Yes Or No: The artifacts we used are open source and canonical.

B5: yes

B5 Elaboration For Yes Or No: Appendix A

B6: yes

B6 Elaboration For Yes Or No: Appendix A

C: yes

C1: yes

C1 Elaboration For Yes Or No: 4

C2: yes

C2 Elaboration For Yes Or No: 4

C3: yes

C3 Elaboration For Yes Or No: 4

C4: yes

C4 Elaboration For Yes Or No: 4

D: no

E: yes

E1: no

E1 Elaboration For Yes Or No: AI assistants only play a basic auxiliary role in experimental programming.