Can Large Language Models Unlock Novel Scientific Research Idea?
Abstract: $\textit{``An idea is nothing more nor less than a new combination of old elements"}$ (Young, J.W.). The widespread adoption of Large Language Models and publicly available ChatGPT has marked a significant turning point in the integration of Artificial Intelligence (AI) into people's everyday lives. This study explores the capability of large language models in generating novel research ideas based on information from research papers. We conduct a thorough examination of 4 LLMs in five domains (e.g., Chemistry, Computer Economics, Medical, and Physics). We found that Claude-2 and GPT-4 generated future research ideas are more aligned with the author's perspective than GPT-3.5 and Gemini. We also found that Claude-2 generates more diverse future research ideas than GPT-4, GPT-3.5, and Gemini 1.0. We further performed a human evaluation of the novelty, relevancy, and feasibility of the generated future research ideas. This investigation offers insights into the evolving role of LLMs in idea generation, highlighting both its capability and limitations. Our work represents a first step toward evaluating and utilizing language models in generating new research ideas. We make our datasets and codes publicly available.
Paper Type: Long
Research Area: NLP Applications
Research Area Keywords: educational applications, applications
Contribution Types: Model analysis & interpretability, Data resources
Languages Studied: English
Submission Number: 839
Loading