Keywords: Multi-hop QA, Knowledge Graphs, KG Embeddings, KGQA, Transformers, Graph Neural Networks
Abstract: 1. Scope of Reproducibility Our work consists of four parts: (1) Reproducing results from Saxena et al.  (2) Adding more experiments by replacing the knowledge graph embedding method (3) and exploring the question embedding method using various transformer models (4) Verifying the importance of Relation Matching (RM) module. Based on the code shared by the authors, we have reproduced the results for the EmbedKGQA method. We have not purposely performed relation matching to validate point-4. 2. Methodology We have used the code provided by Saxena et al.  with some customization for reproducibility. In addition to making the codebase more modular and easy to navigate, we have made changes to incorporate different transformers in the question embedding module. Question-Answering models were trained from scratch as no pre-trained models were available for our particular dataset. The code for this work is available on Github2 . 3. Results We were able to reproduce the Hits@1 to be within ±2.35% of the reported value (in most cases). Anomalies were observed in 3 cases.  In MetaQA-KG-Full (3-hop) dataset  In WebQSP-KG-50 and  WebQSP-KG-Full datasets. From our experiments on the QA model, we have found that a recent transformer architecture, SBERT (Reimers and Gurevych ) produced even better accuracy than the paper. Replacing RoBERTa (Liu et al. ) with SBERT (Reimers and Gurevych ) increased the absolute accuracy by ≈3.4% in half KG case and ≈0.6% in the full KG case. (KG: Knowledge Graph, "≈": Approximately) 4. What was easy As the code was open-sourced, we didn’t have to implement the paper giving us the liberty to customize the codebase to focus on the author’s claim validation, perform extended experiments and explore shared as well as new models. In addition to this, pretrained KG embedding models were shared which helped in the reproduction experiment. 5. What was difficult The lack of comprehensive documentation along with missing comments defining functions/classes/attributes etc. made it laborious to review the code and modify it. In addition to large training times for question answering models, the knowledge graph embeddings also required a significant amount of computing resources. 6. Communication with original authors We had a couple of virtual meetings with Apoorv Saxena, the primary author of EmbedKGQA.
Paper Url: https://openreview.net/forum?id=PUaAgFaMIlaZ