DISAPERE: A Dataset for Discourse Structure in Peer Review DiscussionsDownload PDF

Anonymous

Mar 08, 2022 (edited May 04, 2022)NAACL 2022 Conference Blind SubmissionReaders: Everyone
  • Abstract: At the foundation of scientific evaluation is the labor-intensive process of peer review. This critical task requires participants to consume vast amounts of highly technical text. Prior work has annotated different aspects of review argumentation, but discourse relations between reviews and rebuttals have yet to be examined. We present DISAPERE, a labeled dataset of 20k sentences contained in 506 review-rebuttal pairs in English, annotated by experts. DISAPERE synthesizes label sets from prior work and extends them to include fine-grained annotation of the rebuttal sentences, characterizing their context in the review and the authors' stance towards review arguments. Further, we annotate \textit{every} review and rebuttal sentence. We show that discourse cues from rebuttals can shed light on the quality and interpretation of reviews. Further, an understanding of the argumentative strategies employed by the reviewers and authors provides useful signal for area chairs and other decision makers.
  • Paper Link: https://openreview.net/forum?id=ArWu5lXM5w
  • Paper Type: Long paper (up to eight pages of content + unlimited references and appendices)
  • Dataset: zip
  • Presentation Mode: This paper will be presented in person in Seattle
  • Copyright Consent Signature (type Name Or NA If Not Transferrable): Neha Nayak Kennard
  • Copyright Consent Name And Address: University of Massachusetts Amherst
0 Replies

Loading