Generative Retrieval with Large Language Models

Ye Wang, Rui Xie, Wenxin Hu, Wei Ye, Shikun Zhang

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Generative Retrieval, Large Language Model, Knowledge-sensitive NLP

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We propose a two-stage method that employs the generative retrieval of reference passages by large language models themselves, requiring no additional retrieval models, extra training, or preliminary text chunking.

Abstract: Knowledge-sensitive NLP tasks require access to a large volume of world or domain knowledge. Previous methods all require an extra retrieval model to obtain related reference passages for answering. However, this paper finds that a large language model itself can generate existing passages solely based on the question through constrained decoding, thereby achieving retrieval effects and enhancing prediction. We propose a two-stage method, $\textbf{LLM2GR}$. Specifically, we first prompt the large language model to generate relevant document title identifiers in a constrained manner, then prompt it to generate passages within the document set selected in the first stage, and choose the final reference passage through scoring weighting of the two stages. To speed up the generation retrieval, we only generate shorter prefixes rather than complete passages, then locate them in the document to extract longer, complete reference passages. This method requires no additional retrieval models, no extra training, and no advance text chunking, and can be applied to documents of any length. Experiments on 6 KILT benchmark knowledge-sensitive tasks have verified the effectiveness of our method.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3242