Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents

ACL ARR 2025 May Submission1502 Authors

Abstract:

Research ideation is crucial for scientific progress, but the exponential increase in scientific literature makes it challenging to stay updated and identify impactful directions.
Recent developments in large language models(LLMs) offer a promising avenue to automate this process.
However, existing methods for idea generation either trivially prompt LLMs or expose LLMs to extensive literature without indicating useful information.
Inspired by human research processes, we propose a Chain-of-Ideas~(CoI) agent, an LLM-based agent that organizes relevant literature in a chain structure to effectively mirror the progressive development in a research domain.
This organization helps LLMs better grasp current advancements, thereby improving ideation capabilities.
Further, we present Idea Arena, a protocol for evaluating idea-generation methods from different perspectives, which aligns closely with the preferences of human researchers.
Experiments show that CoI agent consistently outperforms existing methods and matches human quality in idea generation.
Moreover, CoI agent is budget-friendly, requiring only \$0.50 to generate a candidate idea and its experimental design.

Paper Type: Long

Research Area: NLP Applications

Research Area Keywords: Agent,LLM,Idea Generation

Contribution Types: NLP engineering experiment, Publicly available software and/or pre-trained models

Languages Studied: English

Submission Number: 1502