Adaptive Repetition for Mitigating Position Bias in LLM-Based Ranking

Ali Vardasbi, Gustavo Penha, Claudia Hauff, Hugues Bouchard

Abstract: When using LLMs to rank items based on given criteria, or evaluate answers, the order of candidate items can influence the model’s final decision. This sensitivity to item positioning in a LLM’s prompt is known as position bias. Prior research shows that this bias exists even in large models, though its severity varies across models and tasks. In addition to position bias, LLMs also exhibit varying degrees of low repetition consistency, where repeating the LLM call with the same candidate ordering can lead to different rankings. To address both inconsistencies, a common approach is to prompt the model multiple times with different candidate orderings and aggregate the results via majority voting. However, this repetition strategy, significantly increases computational costs.

External IDs:doi:10.1007/978-3-032-12717-4_1