Scientific productivity and practice in the era of Large Language Models

Keigo Kusumegi, Xinyu Yang, Paul Ginsparg, Yian Yin

Keywords: Large Language Models, Productivity, Science of Science

Abstract: Large Language Models (LLMs) are rapidly reshaping scientific research. We analyze these changes in multiple, large-scale datasets with 2.1M preprints, 28K peer review reports, and 246M online accesses to scientific documents. We find: 1) scientists adopting LLMs to draft manuscripts demonstrate a large increase in paper production, ranging from 23.7-89.3\% depending on scientific field and author background, 2) LLM use has reversed the relationship between writing sophistication and paper quality, leading to an influx of manuscripts that are linguistically sophisticated but substantively underwhelming, and 3) LLM adopters read and cite more diverse prior work, including books and younger, less-cited documents. These findings highlight a stunning shift in scientific production that will likely require a change in how journals, funding agencies, and tenure committees evaluate scientific works.

Submission Number: 9