NeurIPS 2021 experiment on author perceptions of review process

Nihar B Shah

A three-part experiment conducted in the review process of NeurIPS 2021 offers striking insights into how researchers perceive, evaluate, and reassess their own work. The study examined three dimensions — authors’ predictions of acceptance, disagreements among co-authors, and how their views changed after the review process — to better understand the psychology of peer review.

The first part investigated how well authors estimate the probability of acceptance of their papers. Authors were asked to estimate their papers’ acceptance chances. The experiment found that the authors consistently overestimated, with median estimates at about 70% compared to the actual 25.8% acceptance rate. The second part focused on how co-authors judged the relative merit of multiple papers they had written together. Each author submitting multiple papers was asked to rank them. Notably, the rankings provided by co-authors on their (multiple) jointly authored papers diverged significantly — about as much as the differences between authors’ opinions and the official review outcomes. The third part examined how the peer-review process changed authors' perceptions of their own papers. Nearly half of the authors changed their opinions of their own papers, and surprisingly, over 30% of authors of rejected papers had a more positive perception of their papers after the review process.

Together, these results underscore the high amount of subjectivity not only in the review process but also in authors' own evaluations. More details and a link to the full paper are available here: [https://blog.ml.cmu.edu/2022/11/22/neurips2021-author-perception-experiment/](https://blog.ml.cmu.edu/2022/11/22/neurips2021-author-perception-experiment/?utm_source=chatgpt.com)