Tree of Clarifications: Answering Ambiguous Questions with Retrieval-Augmented Large Language Models

Gangwoo Kim, Sungdong Kim, Byeongguk Jeon, Joonsuk Park, Jaewoo Kang

Submission Type: Regular Short Paper

Submission Track: Question Answering

Keywords: Question Answering, Large Language Model, Ambiguous QA, Open-domain QA

TL;DR: we introduce Tree of Clarifications: It recursively constructs a tree of interpretations for the ambiguous questions via few-shot prompting leveraging external knowledge; finally to generate a long-form answer.

Abstract:

Questions in open-domain question answering are often ambiguous, allowing multiple interpretations.
One approach to handling them is to identify all possible interpretations of the ambiguous question (AQ) and to generate a long-form answer addressing them all, as suggested by Stelmakh et al., (2022). While it provides a comprehensive response without bothering the user for clarification, considering multiple dimensions of ambiguity and gathering corresponding knowledge remains a challenge.
To cope with the challenge, we propose a novel framework, Tree of Clarifications (ToC):
It recursively constructs a tree of disambiguations for the AQ---via few-shot prompting leveraging external knowledge---and uses it to generate a long-form answer.
ToC outperforms existing baselines on ASQA in a few-shot setup across the metrics, while surpassing fully-supervised baselines trained on the whole training set in terms of Disambig-F1 and Disambig-ROUGE. Code is available at https://github.com/gankim/tree-of-clarifications.

Submission Number: 3121