Embrace Positional Bias: Place Good Documents in Good Positions

ACL ARR 2025 May Submission1329 Authors

Abstract: LLM is unable to treat the information at each position in the prompt fairly, and there is a U-shaped positional bias, which is manifested by paying more attention to the beginning and the end and ignoring the middle, also known as the Lost-in-the-Middle phenomenon. In this paper, we study this phenomenon from the internal state of the model. We examine the effect of different positions on the attention weight of document-level aggregation within the model, both horizontally and vertically, thus reflecting the effect of positional bias on the estimation of document importance in the model. Based on our findings, we propose U-shaped Placement to separate the effects of position and place documents according to positional bias. Combining the U-shaped Placement with the importance estimations of documents within the model, placing good documents in good positions, can improve the model's ability to utilize documents within two iterations. Experimental results prove that our method can outperform other baselines and improve model's ability to utilize documents on various models and datasets. Our codes are submitted with the paper and will be publicly available.

Paper Type: Long

Research Area: Question Answering

Research Area Keywords: interpretability, generalization, open-domain QA

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English

Submission Number: 1329