LLM-based Reranking and Validation of Knowledge Graph Completion
Keywords: Knowledge Graph Completion, Large Language Model, Knowledge Graph Curation
TL;DR: This paper introduces a new workflow that uses LLMs to validate and rerank predictions made by Knowledge Graph Completion systems, reducing human efforts in knowledge graph curation process.
Abstract: In the past decade, Knowledge Graph Completion (KGC), a task aimed at discovering missing knowledge within a knowledge graph by predicting missing links between entities, has garnered significant attention from researchers. Despite significant algorithmic advances, even state-of-the-art KGC models produce predictions with unavoidable inaccuracies. The crucial yet underexplored task of validating these predictions before integration remains predominantly manual, creating a substantial bottleneck in knowledge graph curation pipelines. Unvalidated KGC predictions can propagate errors to downstream knowledge-intensive applications, such as fact-checking, potentially compromising reliability. To address this challenge, we propose a novel workflow leveraging Large Language Models (LLMs) that mimics human validation processes by systematically evaluating KGC predictions made by existing systems through dual verification: analyzing and reranking plausible KGC predictions via internal graph evidence and external knowledge sources. The proposed method leverages token probabilities from LLMs to quantify model confidences and, therefore, provides reranking results of KGC predictions made by existing methods. Our work presents the first large-scale empirical evaluation of using LLMs to post-process KGC predictions on standard benchmarks. Experimental results demonstrate that our approach can significantly reduce human validation efforts while maintaining high factual accuracy standards in knowledge graph curation.
Submission Number: 8
Loading