Attention Weights as an Indicator: Analyzing and Improving Document Utilization in Retrieval-Augmented Generation

ACL ARR 2026 January Submission4374 Authors

Keywords: Attention Weights, Document Utilization, Retrieval-Augmented Generation

Abstract: The generation of Retrieval-Augmented Generation (RAG) models is affected by factors such as the quality and order of input documents, indicating that their ability to utilize documents remains underdeveloped. This ability encompasses not only identifying useful documents from inputs but also minimizing positional bias and filtering irrelevant documents. To achieve this, key challenges include the model's internal estimation of document importance and positional bias. In this paper, we conduct a comprehensive study on the properties of attention weights, examining the impact of factors like aggregation methods, document quality, document position, token type, and so on. Based on our findings, we propose strategies to enhance document utilization from three perspectives: document ranking, placement, and filtering. Comprehensive experiments show that our method outperforms baselines and improves document utilization effectiveness in a training-free manner. Our code is submitted with the paper and will be publicly available.

Paper Type: Long

Research Area: Question Answering

Research Area Keywords: interpretability, generalization,open-domain QA

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English

Submission Number: 4374