Evaluating the Robustness of Machine Reading Comprehension Models to Low Resource Entity Renaming

Clemencia Siro, Tunde Oluwaseyi Ajayi

Keywords: Robustness, machine reading comprehension models, african entities

TL;DR: Evaluating the robustness of MRC models to entity renaming at test-time with entities of African-origin on the SQuAD dataset.

Abstract:

Question answering (QA) models have shown compelling results in the task of
Machine Reading Comprehension (MRC). Recently these systems have proved to
perform better than humans on held-out test sets of datasets e.g. SQuAD, but their
robustness is not guaranteed. The QA model’s brittleness is exposed when evaluated on adversarial generated examples by a performance drop. In this study, we
explore the robustness of MRC models to entity renaming, with entities from low-resource regions such as Africa. We propose EntSwap, a method for test-time perturbations, to create a test set whose entities have been renamed. In particular, we
rename entities of type: country, person, nationality, location, organization, and
city, to create AfriSQuAD2. Using the perturbed test set, we evaluate the robust-
ness of three popular MRC models. We find that compared to base models, large
models perform well comparatively on novel entities. Furthermore, our analysis
indicates that entity type person highly challenges the model performance.