Evaluating the Knowledge Base Completion Potential of GPT

Blerta Veseli, Simon Razniewski, Jan-Christoph Kalo, Gerhard Weikum

Submission Type: Regular Short Paper

Submission Track: Theme Track: Large Language Models and the Future of NLP

Submission Track 2: Information Extraction

Keywords: knowledge base completion, knowledge graphs, probing language models, evaluation

TL;DR: A realistic assessment of the potential of a large language models (GPT-3, GPT-35, GPT-4) for knowledge base completion.

Abstract:

Structured knowledge bases (KBs) are an asset for search engines and other applications but are inevitably incomplete.
Language models (LMs) have been proposed for unsupervised knowledge base completion (KBC), yet, their ability to do this at scale and with high accuracy remains an open question.
Prior experimental studies mostly fall short because they only evaluate on popular subjects, or sample already existing facts from KBs.
In this work, we perform a careful evaluation of GPT's
potential to complete the largest public KB: Wikidata.
We find that, despite their
size and capabilities,
models like GPT-3, ChatGPT and GPT-4 do
not achieve fully convincing results on this task.
Nonetheless, it provides solid improvements over earlier approaches with smaller LMs.
In particular, we show that it is feasible
to extend Wikidata by 27M facts at 90% precision.

Submission Number: 1382