MLMLM: Link Prediction with Mean Likelihood Masked Language Model

Amal Zouaq

Abstract:

Knowledge Bases (KBs) are easy to query, verifiable, and interpretable. They however scale
with man-hours and high-quality data. Masked
Language Models (MLMs), such as BERT,
scale with computing power as well as unstructured raw text data. The knowledge contained
within these models is however not directly interpretable. We propose to perform link prediction with MLMs to address both the KBs
scalability issues and the MLMs interpretability issues. By committing the knowledge embedded in MLMs to a KB, it becomes interpretable. To do that we introduce MLMLM,
Mean Likelihood Masked Language Model,
an approach comparing the mean likelihood of
generating the different entities to perform link
prediction in a tractable manner. We obtain
State of the Art (SotA) results on the WN18RR
dataset and SotA results on the Precision@1
metric on the WikidataM5 inductive and transductive setting. We also obtain convincing results on link prediction on previously unseen
entities, making MLMLM a suitable approach
to introducing new entities to a KB.