Fair comparison of knowledge graphs for question answering

Anonymous

Abstract: Knowledge graphs are commonly used as sources of information in question answering. Models often combine pre-trained text encoders with a graph encoder to use this information to increase accuracy. However, the way that these two types of model interact is not clear. Here we show that, when provided with graph information for a random question, two recent models exhibit no significant change in performance. These models cannot therefore be used to obtain graph-structured explanations, or to compare the relevance of a particular knowledge graph to a dataset. We perform two model ablations and show that the resulting model is more responsive to variation in graph input, and so can be used for gathering explanations and measuring KG-dataset fit. We also show that uncontrollable nondeterminism can cause significant changes in results, and highlight the importance of statistical testing of these models.