Evaluating the Projectivity of Presupposition Triggers in Various Entailment-Canceling Environments

Anonymous

Abstract: Previous studies investigate the ability of models to make pragmatic inferences using presupposition triggers. However, although projection of presuppositions can vary depending on the combination of triggers and environments, they evaluate the performance of models without human baseline, or include only negative sentences as entailment-canceling environments. To evaluate inferences with presupposition triggers, it is necessary to solicit human judgments as a baseline for model evaluation and use various types of entailment-canceling environments. In this study, we introduce a template-based natural language inference dataset called Projectivity of Presupposition Triggers (PPT), which includes 9,800 sentence pairs crossed with six types of presupposition triggers and four types of syntactic environments. Analyzing judgements from 283 people on a subset of the dataset, we find that humans take most presupposition patterns as projective, but the projectivity varies depending on the combination of triggers and environments. In contrast, models judge some patterns as non-projective, indicating that the ability of the models to process presuppositions may not be human-like.This result highlights that researchers working on model evaluation and dataset creation need to take extra care of the combination of presupposition triggers and environments where they are embedded.

Paper Type: long