PlanRAG: A Plan-then-Retrieval Augmented Generation for Generative Large Language Models as Decision Makers

Anonymous

Abstract: In this paper, we conduct a study to utilize LLMs as a solution for decision making that requires complex data analysis. We define Decision QA as the task of answering the best decision, $d_{best}$, for a given natural language question $Q$ and data $D$.There is no benchmark which can examine Decision QA, we propose Decision QA benchmark, DQA, composed of the locating and building scenarios constructed from two video games (Europa Universalis IV and Victoria 3) which have almost the same goal as Decision QA. To address Decision QA effectively, we also propose a new RAG technique called the iterative plan-then-retrieval augmented generation (PlanRAG). In our PlanRAG, the PlanRAG-based LM generates the plan for data analysis in the first planning step, and the retriever generates the queries for data analysis in the second retrieving step. The proposed method outperforms the state-of-the-art iterative RAG method by 12.4% in the locating scenario and by 1.8% in the building scenario, respectively.

Paper Type: long

Research Area: Generation

Contribution Types: NLP engineering experiment, Data resources

Languages Studied: English

Consent To Share Submission Details: On behalf of all authors, we agree to the terms above to share our submission details.