ExpMRC: Explainability Evaluation for Machine Reading Comprehension

Yiming Cui, Ting Liu, Wanxiang Che, Zhigang Chen, Shijin Wang

Keywords: reading comprehension, question answering, explainable AI

TL;DR: We propose ExpMRC, a dataset for explainable machine reading comprehension.

Abstract: Achieving human-level performance on some Machine Reading Comprehension (MRC) datasets is no longer challenging with the help of powerful Pre-trained Language Models (PLMs). However, it is necessary to provide both answer prediction and its explanation to further improve the MRC system's reliability, especially for real-life applications. In this paper, we propose a new benchmark called ExpMRC for evaluating the explainability of the MRC systems. ExpMRC contains four subsets, including SQuAD, CMRC 2018, RACE$^+$, and C$^3$, with additional annotations of the answer's evidence. The MRC systems are required to give not only the correct answer but also its explanation. We use state-of-the-art pre-trained language models to build baseline systems and adopt various unsupervised approaches to extract evidence without a human-annotated training set. The experimental results show that these models are still far from human performance, suggesting that the ExpMRC is challenging.

URL: https://github.com/ymcui/expmrc

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2105.04126/code)