Allocate Marginal Reviews to Borderline Papers Using LLM Comparative Ranking

Elliot L Epstein, Rajat Vadiraj Dwaraknath, John Winnicki, Thanawat Sornwanee

Keywords: Peer review systems, Reviewer assignment, Resource allocation, Large language models, Comparative ranking, Bradley–Terry model

TL;DR: We propose LLM-based comparative ranking before human review to identify borderline papers and allocate marginal reviewer capacity where it has the highest expected impact on decision quality.

Abstract:

This paper argues that large ML conferences should allocate marginal review capacity primarily to papers near the acceptance boundary, rather than spreading extra reviews via random or affinity-driven heuristics.
We propose using LLM-based comparative ranking (via pairwise comparisons and a Bradley--Terry model) to identify a borderline band \emph{before} human reviewing and to allocate \emph{marginal} reviewer capacity at assignment time.
Concretely, given a venue-specific minimum review target (e.g., 3 or 4), we use this signal to decide which papers receive one additional review (e.g., a 4th or 5th), without conditioning on any human reviews and without using LLM outputs for accept/reject.
We provide a simple expected-impact calculation in terms of (i) the overlap between the predicted and true borderline sets ($\rho$) and (ii) the incremental value of an extra review near the boundary ($\Delta$), and we provide retrospective proxies to estimate these quantities.

Track: Long Paper

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 99