Go to ACL ARR 2026 January homepageARR-Dv2: New Data from ACL Rolling Review and its Potential to Improve Reviewing in the ACL Community
Keywords: NLP Applications, Resources and Evaluation, Peer review, Dataset
Abstract: Peer review is a central part of the scientific publication cycle, and the peer review service has long been an honor and obligation that researchers gladly fulfill. With the considerable increase in submissions, this aspect of voluntary contribution to the community can become a burden due to the increased workload. We present a new dataset to support AI-assisted and human studies of the peer reviewing process, based on the ARR Open Review Platform. Unlike previous similar data collections, ours, ARR-Dv2, not only contains the reviews but also author-reviewer discussions (aka rebuttal phase) and meta-reviews. We present statistical analyses of the data with respect to the overall score and the correlations among the overall score, confidence, and soundness scores. Additionally, we extend the task of Review Score Prediction also to include rebuttal data and analyze its effect on score prediction. Finally, we introduce a new task, Meta Review Score Prediction, based on a set of up to three reviews rather than a single review. Our initial results in a zero-shot setting indicate that the rebuttal data adds valuable information to the score prediction and enables reliable predictions.
Paper Type: Long
Research Area: Resources and Evaluation
Research Area Keywords: NLP Applications , Resources and Evaluation, Peer review , Dataset
Contribution Types: Data resources, Data analysis
Languages Studied: English
Submission Number: 6491
Loading