Counterfactual Variable Control for Robust and Interpretable Question Answering
Abstract: Deep learning based question answering (QA) models are neither robust nor interpretable in many cases. For example, a multiple-choice QA model, tested without any input of question, is surprisingly ``capable'' to predict most of correct answers. In this paper, we inspect such ``shortcut capability'' of the QA model using causal inference. We find the crux behind is the shortcut correlation (learned in the model), \eg, simply word alignment between passage and options. To address the issue, we propose a novel inference approach called Counterfactual Variable Control (CVC) that explicitly mitigates any shortcut correlations and preserves only comprehensive reasoning to do robust QA. To enable CVC inference, we first leverage a multi-branch network architecture based on which we disentangle shortcut correlations and comprehensive reasoning in the trained model. Then, we introduce two variants of CVC inference approach to capture only the causal effect of comprehensive reasoning as the model prediction. To evaluate CVC, we conduct extensive experiments using three neural network backbones (BERT-base, BERT-large and RoBERTa-large) on both multi-choice and span-extraction QA benchmarks (MCTest, DREAM, RACE and SQuAD). Our results show that CVC can achieve consistently high robustness against various adversarial attacks in QA tasks, and its results are easy to interpret.
Submission Length: Regular submission (no more than 12 pages of main content)
Assigned Action Editor: ~Manzil_Zaheer1
Submission Number: 731
Loading