AttentionRetriever: Attention Layers are Secretly Long Document Retrievers

ACL ARR 2026 January Submission8846 Authors

Keywords: Long Document Retrieval, Attention Interpretability

Abstract: Retrieval augmented generation (RAG) has been widely adopted to help Large Language Models (LLMs) to process tasks involving long documents. However, existing retrieval models are not designed for long document retrieval and fail to address several key challenges of long document retrieval, including context-awareness, causal dependence, and scope of retrieval. In this paper, we proposed AttentionRetriever, a novel long document retrieval model that leverages attention mechanism and entity-based retrieval to build context-aware embeddings for long document and determine the scope of retrieval. With extensive experiments, we found AttentionRetriever is able to outperform existing retrieval models on long document retrieval datasets by a large margin while remaining as efficient as dense retrieval models.

Paper Type: Long

Research Area: Information Extraction and Retrieval

Research Area Keywords: Information Retrieval and Text Mining, Information Extraction, Interpretability and Analysis of Models for NLP

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English

Submission Number: 8846