Ground-Truth Subgraphs for Better Training and Evaluation of Knowledge Graph Augmented LLMs

TMLR Paper9166 Authors

Abstract:

Retrieval of information from graph-structured knowledge bases represents a promising direction for improving the factuality of LLMs. While various solutions have been proposed, a comparison of methods is difficult due to the lack of challenging QA datasets with ground-truth targets for graph retrieval.
We present SynthKGQA, an LLM-powered framework for generating high-quality Knowledge Graph Question Answering datasets from any Knowledge Graph, providing the full set of ground-truth facts in the KG to reason over questions. To demonstrate its utility, we apply SynthKGQA to Wikidata to generate GTSQA. This new dataset is specifically designed to test zero-shot generalization with respect to unseen graph structures and relation types, enabling us to analyze the abilities and limitations of SOTA graph retrieval approaches at an unprecedented level of granularity. We also show that KG retrievers trained on GTSQA can transfer to human-curated benchmarks, and that the ground-truth subgraphs produced by SynthKGQA provide a better training supervision signal than previously-used heuristics.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Tian_Li1

Submission Number: 9166