An Annotation Protocol for Collecting User-Generated Counter-Arguments Using Crowdsourcing

Paul Reisert, Gisela Vallejo, Naoya Inoue, Iryna Gurevych, Kentaro Inui

Abstract: Constructive feedback is important for improving critical thinking skills. However, little work has been done to automatically generate such feedback for an argument. In this work, we experiment with an annotation protocol for collecting user-generated counter-arguments via crowdsourcing. We conduct two parallel crowdsourcing experiments, where workers are instructed to produce (i) a counter-argument, and (ii) a counter-argument after identifying a fallacy. Our analysis indicates that we can collect counter-arguments that are useful as constructive feedback, especially when workers are first asked to identify a fallacy type.