SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Tasks

Yilun Zhao, Kaiyan Zhang, Tiansheng Hu, Sihong Wu, Ronan Le Bras, Yixin Liu, Xiangru Tang, Joseph Chee Chang, Jesse Dodge, Jonathan Bragg, Chen Zhao, Hannaneh Hajishirzi, Doug Downey, Arman Cohan

Keywords: Literature Review Generation, AI for Research, Benchmark and Evaluation, Foundation Model

Abstract:

We present SciArena, an open and collaborative platform for evaluating foundation models on scientific literature-grounded tasks.
Unlike traditional benchmarks for scientific literature understanding and synthesis, SciArena engages the research community directly, following the Chatbot Arena evaluation approach of community voting on model comparisons.
By leveraging collective intelligence, SciArena offers a community-driven evaluation of model performance on open-ended scientific tasks that demand literature-grounded, long-form responses.
The platform currently supports 44 open-source and proprietary foundation models and has collected over 19,000 votes from human researchers across diverse scientific domains.
Our analysis of the data collected so far confirms its high quality.
We discuss the results and insights based on the model ranking leaderboard.
To further promote research in building model-based automated evaluation systems for literature tasks, we release SciArena-Eval, a meta-evaluation benchmark based on our collected preference data.
The benchmark measures the accuracy of models in judging answer quality by comparing their pairwise assessments with human votes.
Our experiments highlight the benchmark’s challenges and emphasize the need for more reliable automated evaluation methods.

Dataset URL: https://huggingface.co/datasets/yale-nlp/SciArena

Code URL: https://github.com/yale-nlp/SciArena

Primary Area: Evaluation (e.g., data collection methodology, data processing methodology, data analysis methodology, meta studies on data sources, extracting signals from data, replicability of data collection and data analysis and validity of metrics, validity of data collection experiments, human-in-the-loop for data collection, human-in-the-loop for data evaluation)

Flagged For Ethics Review: true

Submission Number: 1082