Agent Reviewers: Domain-specific Multimodal Agents with Shared Memory for Paper Review

Kai Lu, Shixiong Xu, Jinqiu Li, Kun Ding, Gaofeng Meng

TL;DR: We present Agent Reviewers, an LLM-based multi-agent system for peer review, enhanced by multimodal feedback and a shared memory of prior papers.

Abstract: Feedback from peer review is essential to improve the quality of scientific articles. However, at present, many manuscripts do not receive sufficient external feedback for refinement before or during submission. Therefore, a system capable of providing detailed and professional feedback is crucial for enhancing research efficiency. In this paper, we have compiled the largest dataset of paper reviews to date by collecting historical open-access papers and their corresponding review comments and standardizing them using LLM. We then developed a multi-agent system that mimics real human review processes, based on LLMs. This system, named Agent Reviewers, includes the innovative introduction of multimodal reviewers to provide feedback on the visual elements of papers. Additionally, a shared memory pool that stores historical papers' metadata is preserved, which supplies reviewer agents with background knowledge from different fields. Our system is evaluated using ICLR 2024 papers and achieves superior performance compared to existing AI-based review systems. Comprehensive ablation studies further demonstrate the effectiveness of each module and agent in this system.

Lay Summary:

Peer review is the process where experts give feedback on scientific papers before they are published. This feedback helps authors improve their work and ensures the quality of research. However, many researchers—especially early-career ones—struggle to get useful feedback due to the increasing number of papers and limited time from human reviewers.
To help solve this, we created a system called Agent Reviewers that uses artificial intelligence to simulate how real reviewers work. Our system includes multiple AI “agents” that specialize in different topics, and even one that can evaluate images and charts in a paper—something most AI tools can’t do. These agents read a paper, discuss with each other, and produce thoughtful comments and suggestions.
To support this system, we also gathered and standardized the largest public dataset of scientific reviews. We tested our tool on hundreds of real research papers and found that it provides more helpful and diverse feedback than existing AI systems. We hope Agent Reviewers can help researchers improve their papers and make the peer review process more accessible and effective for everyone.

Link To Code: https://github.com/AReviewers/AgentReviewers

Primary Area: Applications->Language, Speech and Dialog

Keywords: Peer Review Automation; Multimodal Review System; Multi-agent System; Shared Memory Pool;

Submission Number: 9153