FigurativeQA: A Test Benchmark for Figurativeness Comprehension for Question Answering

Geetanjali Rakshit, Jeffrey Flanigan

Abstract:

Figurative language is widespread in human language (Lakoff and Johnson, 2008), posing potential challenges in NLP applications. In
this paper, we investigate the effect of figurative language on the task of question answering (QA). We construct FigurativeQA, a test set of
400 yes-no questions with figurative and nonfigurative contexts, extracted from product reviews and restaurant reviews. We demonstrate that a state-of-the-art RoBERTa QA model has considerably lower performance in question answering when the contexts are figurative rather than literal, indicating a gap in current models. We propose a general method for improving the performance of QA models by converting the figurative contexts into non-figurative by prompting GPT-3, and demonstrate its effectiveness. Our results indicate a need for
building QA models infused with figurative language understanding capabilities.