Construction Artifacts in Metaphor Identification Datasets

Joanne Boisson, Luis Espinosa-Anke, Jose Camacho-Collados

Submission Type: Regular Short Paper

Submission Track: Resources and Evaluation

Submission Track 2: Linguistic Theories, Cognitive Modeling, and Psycholinguistics

Keywords: metaphors, resources, bias

TL;DR: We show how existing metaphor identification datasets can be gamed by fully ignoring the context in which the potential metaphorical expression occurs.

Abstract: Metaphor identification aims at understanding whether a given expression is used figuratively in context. However, in this paper we show how existing metaphor identification datasets can be gamed by fully ignoring the potential metaphorical expression or the context in which it occurs. We test this hypothesis in a variety of datasets and settings, and show that metaphor identification systems based on language models without complete information can be competitive with those using the full context. This is due to the construction procedures to build such datasets, which introduce unwanted biases for positive and negative classes. Finally, we test the same hypothesis on datasets that are carefully sampled from natural corpora and where this bias is not present, making these datasets more challenging and reliable.

Submission Number: 4555