When to Retrieve? Teaching LLMs to Utilize Information Retrieval Effectively

ACL ARR 2024 August Submission226 Authors

Abstract: Recently, systems that combine Information Retrieval (IR) with Large Language Models (LLMs), such as RAG, have demonstrated remarkable capabilities in question answering by integrating external context. However, the optimal strategy for question answering does not always involve retrieving external information; it often involves leveraging the LLM’s own parametric memory. In this paper, we demonstrate how LLMs can be effectively trained to determine when additional context is necessary and to utilize an off-the-shelf IR system accordingly. We propose a tailored training approach where LLMs, using open-domain question answering datasets, learn to generate a special token, $\langle$RET$\rangle$, when they do not know the answer to a question. Our evaluation of the Adaptive Retrieval LLM (Adapt-LLM) on the PopQA dataset showcases improvements over the same LLM under three configurations: (i) retrieving information for all questions, (ii) relying solely on the LLM's parametric memory, and (iii) using a popularity threshold to decide when to use a retriever.

Paper Type: Long

Research Area: Question Answering

Research Area Keywords: Question Answering, Generation, Information Retrieval and Text Mining, Language Modeling

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Publicly available software and/or pre-trained models, Data analysis

Languages Studied: English

Submission Number: 226