Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents

ACL ARR 2025 February Submission853 Authors

Abstract: Research ideation is a critical step for scientific research. However, given the exponential increase in scientific literature, researchers are difficult to stay current with recent advances and identify meaningful research directions. Recent developments in large language models(LLMs) suggest a promising avenue to automate this process. However, existing methods for idea generation either trivially prompt LLMs or expose LLMs to extensive literature without indicating useful information. Inspired by the human research process, we propose a Chain-of-Ideas(CoI) agent, an LLM-based agent that organizes relevant literature in a chain structure to effectively mirror the progressive development in a research domain.This organization facilitates LLMs to capture current research advancements, thereby enhancing their ideation capabilities. Furthermore, we propose Idea Arena, an evaluation protocol for evaluating idea-generation methods from different perspectives, which aligns closely with the preferences of human researchers.Experiments show that the CoI agent consistently outperforms other methods and shows comparable quality as humans in idea generation. Moreover, our CoI agent is budget-friendly, necessitating only \$0.50 to generate a candidate idea and its corresponding experimental design.

Paper Type: Long

Research Area: NLP Applications

Research Area Keywords: Agent,LLM,Idea Generation

Contribution Types: NLP engineering experiment, Publicly available software and/or pre-trained models

Languages Studied: English

Submission Number: 853