emEval-2022 Task 3: PreTENS-Evaluating Neural Networks on Presuppositional Semantic Knowledge

Roberto Zamparelli, Shammur Absar Chowdhury, Dominique Brunato, Cristiano Chesi, Felice Dell'Orletta, Md. Arid Hasan, Giulia Venturi

Abstract:

We report the results of the SemEval 2022 Task
3, PreTENS, on evaluation the acceptability
of simple sentences containing constructions
whose two arguments are presupposed to be
or not to be in an ordered taxonomic relation.
The task featured two sub-tasks articulated as:
(i) binary prediction task and (ii) regression
task, predicting the acceptability in a continu-
ous scale. The sentences were artificially gen-
erated in three languages (English, Italian and
French). 21 systems, with 8 system papers
were submitted for the task, all based on var-
ious types of fine-tuned transformer systems,
often with ensemble methods and various data
augmentation techniques. The best systems
reached an F1-macro score of 94.49 (sub-task1)
and a Spearman correlation coefficient of 0.80
(sub-task2), with interesting variations in spe-
cific constructions and/or languages.