EmbedKGQA: Improving Multi-hop Question Answering over Knowledge Graphs using Knowledge Base Embeddings

Jishnu Jaykumar P, Ashish Sardana

Keywords: Multi-hop QA, Knowledge Graphs, KG Embeddings, KGQA, Transformers, Graph Neural Networks

Abstract:

1. Scope of Reproducibility
Our work consists of four parts: (1) Reproducing results from Saxena et al. [2020] (2) Adding more experiments by
replacing the knowledge graph embedding method (3) and exploring the question embedding method using various
transformer models (4) Verifying the importance of Relation Matching (RM) module. Based on the code shared by
the authors, we have reproduced the results for the EmbedKGQA method. We have not purposely performed relation
matching to validate point-4.

2. Methodology
We have used the code provided by Saxena et al. [2020] with some customization for reproducibility. In addition to
making the codebase more modular and easy to navigate, we have made changes to incorporate different transformers
in the question embedding module. Question-Answering models were trained from scratch as no pre-trained models
were available for our particular dataset. The code for this work is available on Github2
.
3. Results
We were able to reproduce the Hits@1 to be within ±2.35% of the reported value (in most cases).
Anomalies were observed in 3 cases. [1] In MetaQA-KG-Full (3-hop) dataset [2] In WebQSP-KG-50 and [3] WebQSP-KG-Full datasets.
From our experiments on the QA model, we have found that a recent transformer architecture, SBERT (Reimers and
Gurevych [2019]) produced even better accuracy than the paper. Replacing RoBERTa (Liu et al. [2019]) with SBERT
(Reimers and Gurevych [2019]) increased the absolute accuracy by ≈3.4% in half KG case and ≈0.6% in the full KG
case. (KG: Knowledge Graph, "≈": Approximately)

4. What was easy
As the code was open-sourced, we didn’t have to implement the paper giving us the liberty to customize the codebase
to focus on the author’s claim validation, perform extended experiments and explore shared as well as new models. In
addition to this, pretrained KG embedding models were shared which helped in the reproduction experiment.

5. What was difficult
The lack of comprehensive documentation along with missing comments defining functions/classes/attributes etc.
made it laborious to review the code and modify it. In addition to large training times for question answering models,
the knowledge graph embeddings also required a significant amount of computing resources.

6. Communication with original authors
We had a couple of virtual meetings with Apoorv Saxena, the primary author of EmbedKGQA.

Paper Url: https://openreview.net/forum?id=PUaAgFaMIlaZ