Landmark Embedding For Long-Context Retrieval AugmentationDownload PDF

Anonymous

16 Feb 2024ACL ARR 2024 February Blind SubmissionReaders: Everyone
Abstract: Retrieval augmentation is a promising approach to handle long-context language modeling. However, the existing retrieval methods usually work with the chunked context, which is prone to inferior quality of semantic representation and incomplete retrieval of useful information. In this work, we propose a new method for the retrieval augmentation of long-context language modeling, called \textbf{Landmark Embedding}. Our method is characterized by threefold technical contributions. Firstly, we introduce a \textit{chunking-free architecture}, which keeps the long context coherent such that high-quality embeddings can be generated for the fine-grained units within the context. Secondly, we present a \textit{position-aware objective function}, which prioritizes the ultimate boundary for a consecutive span of information. By learning to discriminate such a special position, the useful information can be comprehensively retrieved for the query. Thirdly, we design a novel \textit{multi-stage learning algorithm}, which makes the best use of readily available data and synthetic data for cost-effective training of the landmark embedding. In our experimental study, landmark embedding is able to substantially improve the performance for both LLaMA-2 and ChatGPT in a variety of long-context tasks; meanwhile, it also outperforms the existing retrieval methods with a notable advantage. Our model and source code will be made publicly available.
Paper Type: long
Research Area: Information Retrieval and Text Mining
Languages Studied: English
Preprint Status: We are considering releasing a non-anonymous preprint in the next two months (i.e., during the reviewing process).
A1: yes
A1 Elaboration For Yes Or No: Section 6
A2: yes
A2 Elaboration For Yes Or No: Section 7
A3: yes
A3 Elaboration For Yes Or No: Section 1
B: yes
B1: yes
B1 Elaboration For Yes Or No: Section 2
B2: yes
B3: n/a
B4: n/a
B5: yes
B5 Elaboration For Yes Or No: Appendix A
B6: yes
B6 Elaboration For Yes Or No: Appendix A
C: yes
C1: yes
C1 Elaboration For Yes Or No: Section 4
C2: yes
C2 Elaboration For Yes Or No: Section 4
C3: yes
C3 Elaboration For Yes Or No: Section 4
C4: yes
C4 Elaboration For Yes Or No: Section 4
D: no
D1: n/a
D2: n/a
D3: n/a
D4: n/a
D5: n/a
E: yes
E1: yes
E1 Elaboration For Yes Or No: Appendix A
0 Replies

Loading

OpenReview is a long-term project to advance science through improved peer review with legal nonprofit status. We gratefully acknowledge the support of the OpenReview Sponsors. © 2025 OpenReview