Implementation and evaluation of an additional GPT-4-based reviewer in PRISMA-based medical systematic literature reviews

Assaf Landschaft, Dario Antweiler, Sina Mackay, Sabine Kugler, Stefan Rüping, Stefan Wrobel, Timm Höres, Héctor Allende-Cid

Abstract: Highlights•GPT-4 API as a Complementary Reviewer: Novel integration method for systematic literature reviews, enhancing efficiency and rigor.•Feasibility and Reliability Evaluation: Assessing GPT-4′s potential as a primary screening tool in healthcare information retrieval.•Comprehensive Inter-rater Agreement Analysis: Utilizing Cohen’s kappa to assess agreement between human reviewers, GPT-4, and consensus, employing distinct methodologies for various parameter types.•Full-text Extraction Advancements: Overcoming limitations to extend AI-based reviewer capabilities in evidence synthesis.