Harnessing Multi-role Capabilities of Large Language Models for Open-domain Question Answering
Keywords: Open-domain question answering, large language models, prompt optimization
Abstract: Open-domain question answering (ODQA) stands as a pivotal research spotlight in web mining. The common thread of existing methods for ODQA follows two main paradigms to collect evidence: 1) The \textit{retrieve-then-read} retrieves a set of pertinent documents from an external corpus; and 2) the \textit{generate-then-read} paradigm, which employs large language models (LLMs) to generate relevant documents. Despite both paradigms have their own advantages, a single paradigm cannot take into account multifaceted requirements for evidence. To this end, we propose \model, a generalized framework that formulates the ODQA process into three fundamental steps: query expansion, document selection, and answer generation, which is a novel paradigm combining the superiority of retrieval-based and generation-based evidence. Existing research has verified that LLMs can exhibit their excellent capabilities to accomplish various types of tasks. Therefore, in contrast to previously utilizing specialized models to complete each individual module of ODQA, we instruct LLMs to play multiple roles as generators, rerankers, and evaluators in our unified framework, and integrate them to collaborate each other to jointly enhance the performance of ODQA task. Furthermore, we introduce a novel prompt optimization algorithm to refine the role-play prompts and steer LLMs towards producing higher-quality evidence and more accurate answers. We conduct extensive experiments on three widely used benchmarks: NQ, WebQ, and TriviaQA. Experimental results demonstrate that our \model can achieve the best performance in terms of both answer accuracy and evidence quality, showcasing its potential for advancing ODQA research and applications.
Track: Web Mining and Content Analysis
Submission Guidelines Scope: Yes
Submission Guidelines Blind: Yes
Submission Guidelines Format: Yes
Submission Guidelines Limit: Yes
Submission Guidelines Authorship: Yes
Student Author: Yes
Submission Number: 2130
Loading