Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviews

ACL ARR 2025 May Submission3908 Authors

Abstract:

Peer review underpins scientific progress, but it is increasingly strained by reviewer shortages and growing workloads. Large Language Models (LLMs) can automatically draft reviews now, but determining whether LLM-generated reviews are trustworthy requires systematic evaluation. Researchers have evaluated LLM reviews at either surface-level (e.g., BLEU and ROUGE) or content-level (e.g., specificity and factual accuracy). Yet it remains uncertain whether LLM-generated reviews attend to the same critical facets that human experts weigh---the strengths and weaknesses that ultimately drive an accept-or-reject decision. We introduce a focus-level evaluation framework that operationalizes the focus as a normalized distribution of attention across predefined facets in paper reviews. Based on the framework, we developed an automatic focus-level evaluation pipeline based on two sets of facets: target (e.g., problem, method, and experiment) and aspect (e.g., validity, clarity, and novelty), leveraging 676 paper reviews from OpenReview that consists of 3,657 strengths and weaknesses identified from human experts. The comparison of focus distributions between LLMs and human experts showed that the off-the-shelf LLMs consistently have a more biased focus towards examining technical validity while significantly overlooking novelty assessment when criticizing papers.

Dataset: https://figshare.com/s/d5adf26c802527dd0f62

Paper Type: Long

Research Area: NLP Applications

Research Area Keywords: NLP Applications, Human-Centered NLP, Ethics, Bias, and Fairness, Sentiment Analysis

Contribution Types: Model analysis & interpretability, Data analysis

Languages Studied: English

Keywords: Human-centered NLP, NLP applications

Submission Number: 3908