DivKnowQA: Verifying the Reasoning Ability of LLM Through Open-Domain Question Answering Over Knowledge Base and Text
Keywords: Benchmark, Question Answering, LLM, Retrieval
TL;DR: DIVKNOWQA introduces a challenging benchmark and DETLLM, a novel approach, for improving open-domain complex question answering with Large Language Models.
Abstract: Open-domain complex question answering often breaks down a multi-hop question into single-hop questions, leveraging external knowledge for solutions. Current practices show a pronounced preference for unstructured texts, such as Wikipedia, often overlooking the potential of structured knowledge sources, such as WikiData. Additionally, while existing research has employed external tools to enhance the Large Language Model(LLM)’s capabilities, many tests have been conducted in artificial or toy scenarios. We argue that open-domain complex question answering presents a realistic and intricate challenge for LLM, necessitating the integration of external tools, including retrieval systems and knowledge base engines. In this paper, we present a new benchmark DIVKNOWQA to assess the LLMs’ reasoning skills and tool compatibility. Comprising 940 human-annotated intricate questions, DIVKNOWQA mandates both structured and unstructured knowledge for comprehensive answers. The subpar performance of prevailing SOTA methods, such as DSP and REACT, on our benchmark demonstrates its challenge. Moreover, we introduce our method DETLLM, which incorporates a symbolic language generation tool and a retrieval toolbox, pioneering a new approach to address this challenge. Our data and code will be released
Primary Area: datasets and benchmarks
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 2146
Loading