Leveraging Reference Documents for Zero-Shot Ranking via Large Language Models

TMLR Paper6761 Authors

Abstract: Large language models (LLMs) have proven strong zero-shot rerankers, yet the two dominant paradigms expose a sharp accuracy-efficiency trade-off. Existing methods mainly fall into two categories: Individual-scoring (pointwise) issues $O(n)$ parallel calls but suffers from calibration drift across isolated prompts; Comparative-sorting (pairwise/listwise) alleviates drift via explicit inter-document comparison, but incurs higher-than-linear inference or long single-call latency. To address their limitations, we propose **RefRank**, a reference-anchored framework that marries the throughput of Individual-scoring with the calibration benefits of Comparative-sorting. RefRank prompts the LLM to score each candidate relative to a fixed anchor document harvested from the first-stage top-k list; all candidates are thus implicitly compared through the same anchors while parallelism is preserved. The method is training-free, adds no extra model calls, and keeps complexity at O(n). Across six standard benchmarks and multiple backbones, RefRank significantly outperforms Individual-scoring baselines and surpasses Comparative-sorting competitors with only negligible overhead.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Jonathan_Berant1

Submission Number: 6761