Winograd Schema Challenge: Is it the Reporting Bias that Makes it Difficult or the Demanding Nature of the Reasoning Systems?


Nov 17, 2018 AKBC 2019 Conference Blind Submission readers: everyone Show Bibtex
  • Keywords: Winograd Schema Challenge, Question Answering, Commonsense Reasoning
  • TL;DR: We present a knowledge base for the Winograd Schema Challenge and a reasoning algorithm which converts the Winograd Schema Challenge into a Natural Language Inference problem.
  • Abstract: The Winograd Schema Challenge contains multiple choice questions where deciding the correct choice requires the use of commonsense knowledge. The needed knowledge is not mentioned in the questions. So answering a question requires extraction of ``useful" knowledge and reasoning with the extracted knowledge. Due to reporting bias (people rarely state the obvious) the extraction of commonsense knowledge is a nontrivial task and on top of it if the reasoning system is naive then the burden on knowledge extraction increases heavily. This results in a low recall which ultimately results in a poor performance on the original dataset. However by performing a careful search over the web, we observe that it is possible to collect useful knowledge (a sentence) for each question. Existing knowledge based solvers however could not utilize the extracted knowledge well. The task we then focus on is how to develop a sophisticated reasoning system which can better utilize the available knowledge and reduce the burden on knowledge extraction. We present one such method in this work which takes as input a Winograd Schema question and a knowledge sentence and then converts the Winograd Schema problem into a Natural Language Inference (Textual Entailment) problem with the help of semantic role labelling to predict the answer. This approach of reasoning obtains significant improvement compared to the existing knowledge based Winograd Schema solvers. The manually extracted knowledge base and the system is publicly available at
  • Archival status: Non-Archival
  • Subject areas: Natural Language Processing, Question Answering, Reasoning
0 Replies