How Effective is Your Rebuttal? Identifying Causal Models from the OpenReview System

Loka Li, Ibrahim Aldarmaki, Minghao Fu, Wong Yu Kang, Yunlong Deng, Qiang Huang, Jing Yang, Jin Tian, Guangyi Chen, Kun Zhang

Keywords: Open review, latent causal model, representation learning

TL;DR: This paper analyzed comprehensive data from the OpenReview system to examine how various rebuttal characteristics causally influence post-rebuttal score changes.

Abstract: The peer review process is central to scientific publishing, with the rebuttal phase offering authors a critical opportunity to address reviewers’ concerns. Yet the causal mechanisms underlying rebuttal effectiveness, particularly how author responses influence final review decisions, remain unclear. To uncover the mechanisms driving rebuttal effectiveness, we model it as a causal representation learning (CRL) problem. Using data from the OpenReview system for ICLR submissions, we examine how rebuttal characteristics of both reviewers and authors causally affect post-rebuttal rating changes. We introduce a weakly supervised disentangled CRL framework that leverages review subscores (e.g., openness, clarity, directness) as concept-level supervision. Theoretically, we establish identifiability conditions for latent variables across multiple distributions, showing that human-interpretable concepts can be recovered under mild assumptions. Empirically, our results uncover distinct causal patterns governing successful rebuttals, revealing how specific strategies differentially influence review criteria. These findings provide actionable guidance for authors in crafting effective rebuttals, while offering broader implications for transparency, fairness, and efficiency in the peer review process.

Submission Number: 48