TASA: Twin Answer Sentences Attack for Adversarial Context Generation in Question Answering

Anonymous

Abstract: We present Twin Answer Sentences Attack (TASA), a novel question answering (QA) adversarial attack method that produces fluent and grammatical adversarial contexts while maintaining its gold answers. Despite phenomenal progresses on general adversarial attacks, few works have investigated the vulnerability and adversarial attack specifically for QA. In this work, we first investigate the biases in the existing models and discover that they heavily rely on keyword matching and ignore the relevant entities from the question. TASA explores the two biases above and attacks the target model in two folds: (1) lowering the model's confidence on the gold answer with a perturbed answer sentence; (2) misguiding the model towards a wrong answer with a distracting answer sentence. Equipped with designed beam search and filtering methods, TASA is able to attack the target model efficiently while sustaining the quality of contexts. Extensive experiments on four QA datasets and human evaluations demonstrate that TASA generates substantial-high-quality attacks than existing textual adversarial attack methods.

Paper Type: long

Consent To Share Data: yes