Self-Aware AI Review Bias Detection: Enabling Real-Time Bias Identification in AI-Generated Scientific Reviews

Keywords: AI, Bias, mitigation, review, self-aware, scientific evaluation, LLM

TL;DR: We developed the first systematic framework to detect and reduce AI bias in scientific peer review revealing critical model-dependent effectiveness of self-aware AI systems.

Abstract: As AI systems increasingly participate in scientific peer review, understanding and mitigating their inherent biases becomes critical for maintaining research integrity. We present the first systematic investigation of self-aware bias detection in AI-generated scientific reviews, where AI reviewers identify and correct their own biases in real-time during review generation. Our framework analyzes five key bias types: position bias, length bias, negativity bias, self-enhancement bias, and domain familiarity bias. Through controlled experiments across four state-of-the-art language models (GPT-4o, Claude-Sonnet-4, Llama-3.1-8B, Mistral-7B) on 6 scientific papers per model, we demonstrate significant bias reduction with Claude-Sonnet-4 achieving 36.2\% bias reduction (p < 0.001, Cohen's d = 3.62) and 85.6\% confidence improvement. Our statistical analysis with Bonferroni correction confirms robust results across all models with large effect sizes (d > 1.77). This work establishes the first quantitative framework for AI reviewer self-awareness and provides a foundation for developing more reliable AI-assisted peer review systems.

Submission Number: 47