From Papers To Peers: LLM-based Algorithm For Selecting Reviewers
Keywords: scientific texts, reviewer selection, large language models, text embeddings, recommendation systems
TL;DR: A two-stage LLM-based system automates reviewer selection by matching paper content with researchers' expertise through embedding comparisons and language model re-ranking.
Abstract: The rapid growth in the number of annually published scientific papers places a significant burden on the editors of scientific journals and conference organizers. In particular, quickly selecting relevant reviewers becomes difficult. Automation of this process is hampered by the lack of publicly available information on reviewers of already published articles in accordance with the requirements of double-blind peer review. In this paper, we present an algorithm for automatic two-stage selection of reviewers based on the title, abstract, and optional Universal Decimal Classification (UDC) code of the reviewed article. The first stage involves the selection and filtering of relevant candidates from the database of authors of scientific articles by comparing neural text embeddings of the reviewed article with candidates' published papers, and the second stage involves re-ranking the selected candidates using a large language model. Since information on real reviewers of articles cannot be collected, we assess the quality of generation using several indirect automatically calculated metrics: the presence of the authors of the original article among the top-k recommended candidates and the presence of a common UDC code between the original article and the articles of the candidates. We also present the outcomes of human evaluation of the recommended reviewers and analyze the strengths and weaknesses of our algorithm through expert assessment. Our results demonstrate that the proposed algorithm can effectively serve as a highly qualified assistant in the reviewer selection process, significantly reducing the manual effort required from editors.
Submission Number: 20
Loading