CR-Bench: Evaluating the Real-World Utility of AI Code Review Agents

Kristen Pereira, Neelabh Sinha, Rajat Ghosh, Debojyoti Dutta

Keywords: Dataset, Evaluation, Automated Code Review, LLM Agents, Software Reliability, Benchmarking, Agentic Reasoning

TL;DR: We introduce CR-Bench and CR-Evaluator to enable fine-grained evaluation of LLM-based code review agents, revealing a key trade-off where aggressively finding issues increases false positives and limits real-world usefulness.

Abstract: Recent advances in frontier large language models have enabled code review agents that operate in open-ended, reasoning-intensive settings. However, the lack of standardized benchmarks and granular evaluation protocols makes it difficult to assess behavior of code review agents beyond coarse success metrics, particularly for tasks where false positives are costly. To address this gap, we introduce CR-Bench, a benchmarking dataset, and CR-Evaluator, a fine-grained evaluation pipeline for code review agents. Using these tools, we conduct a preliminary study evaluating both a single-shot agent and a Reflexion-based agent across two frontier models. We find that code review agents can exhibit a low signal-to-noise ratio when designed to identify all hidden issues, obscuring true progress and developer productivity when measured solely by resolution rates. Our analysis identifies the hidden trade-off between issue resolution and spurious findings, revealing a frontier that constrains effective agent design. Together, CR-Bench and CR-Evaluator provide a timely foundation for studying and developing code review agents as LLM-based systems transition from controlled benchmarks to real-world software engineering workflows.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 149