Distance between Relevant Information Pieces Causes Bias in Long-Context LLMs

ACL ARR 2024 December Submission1373 Authors

Abstract: Positional bias in large language models hinders their ability to effectively process long inputs. A prominent example is the "lost in the middle" phenomenon, where LLMs struggle to utilize relevant information situated in the middle of the input. While prior research primarily focuses on single pieces of relevant information, real-world applications often involve multiple relevant information pieces. To bridge this gap, we present LongPiBench, a benchmark designed to assess positional bias involving multiple pieces of relevant information. It includes various tasks and input lengths. Thorough experiments are conducted with three commercial and six open-source models. These experiments reveal that while most current models are more robust against the "lost in the middle" issue, there also exist noticeable biases related to the spacing of relevant information pieces. These findings highlight the importance of evaluating and reducing positional biases for long-context LLMs.

Paper Type: Short

Research Area: Resources and Evaluation

Research Area Keywords: benchmarking, evaluation, statistical testing for evaluation

Contribution Types: Model analysis & interpretability, Data resources

Languages Studied: English

Submission Number: 1373