ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows

Qiushi Sun, Zhoumianze Liu, Chang Ma, Zichen Ding, Fangzhi Xu, Zhangyue Yin, Haiteng Zhao, Zhenyu Wu, Kanzhi Cheng, Zhaoyang Liu, Jianing Wang, Qintong Li, Robert Tang, Tianbao Xie, Xiachong Feng, Xiang Li, Ben Kao, Wenhai Wang, Biqing Qi, Lingpeng Kong et al. (1 additional authors not shown)

Submission Track: Paper Track (up to 8 pages)

Keywords: Computer Use Agents, Scientific Discovery, Multimodal LLM

TL;DR: ScienceBoard provides a realistic environment and a diverse benchmark across six domains, enabling agents to autonomously perform scientific exploration and supporting rigorous evaluation.

Abstract: Large Language Models (LLMs) have extended their impact beyond Natural Language Processing, substantially fostering the development of interdisciplinary research. Recently, various LLM-based agents have been developed to assist scientific discovery progress across multiple aspects and domains. Among these, computer-using agents, capable of interacting with operating systems as humans do, are paving the way to automated scientific problem-solving and addressing routines in researchers’ workflows. Recognizing the transformative potential of these agents, we introduce ScienceBoard, which encompasses two complementary contributions: (i) a realistic, multi-domain environment providing authentic scientific discovery workflows with integrated professional software, where agents can autonomously interact via different interfaces to accelerate complex research tasks and experiments; and (ii) a challenging benchmark of 169 high‑quality, rigorously validated real‑world tasks curated by humans, spanning scientific‑discovery workflows in domains such as biochemistry, astronomy, and geoinformatics. Extensive evaluations demonstrate that, despite some promising results, current agents still fall short of reliably assisting scientists with complex workflows (15% success rate). In‑depth analysis further provides valuable insights for addressing current agent limitations and more effective design principles, paving the way to build more capable agents for scientific discovery. Our codes are available at \url{https://anonymous.4open.science/r/ScienceBoard}.

Camera Ready Modification Summary: (1) New evaluation results added. (2) We corrected several minor typos and incorporated reviewers’ suggestions by adding an additional discussion section in the appendix.

Submission Number: 6