ProPres: Investigating the Projectivity of Presupposition with Various Triggers and Environments

Anonymous

Abstract: A presupposition of a sentence refers to information taken for granted by a speaker and projectivity (e.g., the boy did not shed tears again presupposes \textit{the boy had shed tears before}) is what makes it distinct from entailment. Although the projectivity might vary depending on the combination of presupposition triggers and environments, previous studies evaluate the performance of models without a human baseline or include only negation sentences as an entailment-canceling environment. Hence, it is necessary to both collect human judgments to obtain a baseline and include various environments to investigate the projectivity of presuppositions comprehensively. In this study, we first reevaluate a previous dataset with recent models and humans, then introducing a new dataset, projectivity of presupposition (ProPres), which includes 12k premise--hypothesis pairs crossing six new triggers with five environments. Our large-scale human judgment experiments provide evidence for variable projectivity, but our model evaluation shows that the models do not capture it. This indicates that the models and humans behave differently in the processing of presuppositions. These results cannot be obtained without the human experiments or the combination of various triggers and environments, suggesting that researchers working on the model performance on pragmatic inferences need to take extra care of the annotation process and the combination of various items.

Paper Type: long

Research Area: Discourse and Pragmatics