Angle-QPP: Improving Query Performance Prediction through Large Language Models and Angle Interaction in Complex Vector Space

Xianming Li, Jing Li

Abstract: Query performance prediction (QPP) is a critical task in information retrieval. It estimates retrieval quality for a given query without relying on relevance judgments. While recent approaches have leveraged pretrained (large) language models with binary- or cross-encoder architectures, they struggle to capture subtle semantic differences (nuances that make similar sentences mean different things) between queries and documents in QPP, limiting prediction accuracy. To address this issue, we present Angle-QPP, a novel and efficient binary-encoder QPP approach with three key innovations: (1) the use of Large Language Models (LLMs) of varying scales to learn rich contextual semantics, (2) a contrastive learning warm-up phase to obtain high-quality initial representation quality, and (3) an angle-based interaction mechanism operating in complex embedding space to effectively capture subtle semantic relationships between queries and documents. Comprehensive experiments on TREC DL 2019, 2020, 2021, and 2022 datasets demonstrate that the proposed Angle-QPP significantly outperforms existing methods across all evaluation metrics. Notably, Angle-QPP models with 0.5B, 1.5B, and 3B parameters achieve \(6.4\%\), \(11.2\%\), and \(13.9\%\) absolute improvements in prediction accuracy over the previous state-of-the-art binary-encoder BERT-QPP, respectively. It demonstrates the scalability and effectiveness of the proposed method. Ablation studies confirm the effectiveness of both the angle interaction mechanism and contrastive learning warm-up components. Our analysis further reveals that scaling up LLM size consistently improves QPP performance, providing valuable insights for the design of future query performance prediction systems.

External IDs:doi:10.1145/3793857