HeurekaBench: A Benchmarking Framework for AI Co-scientist

Siba Smarak Panigrahi, Jovana Videnović, Maria Brbic

Keywords: agents, benchmarks for agents, LLMs, single-cell biology

TL;DR: We propose a framework for building benchmarks to evaluate co-scientists in experimental data-driven real-world research and compare existing agents in single-cell biology with architectural modifications to draw insights about agent design choices.

Abstract: LLM-based reasoning models have enabled the development of agentic systems that act as co-scientists, assisting in multi-step scientific analysis. However, evaluating these systems is challenging, as it requires realistic, end-to-end research scenarios that integrate data analysis, interpretation, and the generation of new insights from the experimental data. To address this limitation, we introduce HeurekaBench, a framework to create benchmarks with exploratory, open-ended research questions for experimental datasets. Each such question is grounded in a scientific study and its corresponding code repository, and is created using a semi-automated pipeline that leverages multiple LLMs to extract insights and generate candidate workflows, which are then verified against reported findings. We instantiate the framework in single-cell biology to obtain sc-HeurekaBench benchmark and use it to compare state-of-the-art single-cell agents. We further showcase the benefits of our benchmark for quantitatively analyzing current design choices in agentic systems. We find that the addition of a critic module can improve ill-formed responses for open-source LLM-based agents by up to 22% and close the gap with their closed-source counterparts.

Primary Area: datasets and benchmarks

Submission Number: 24640