DOLPHIN: Moving Towards Closed-loop Auto-research through Thinking, Practice, and Feedback

Jiakang Yuan, Xiangchao Yan, Shiyang Feng, Bo Zhang, Tao Chen, Botian Shi, Wanli Ouyang, Yu Qiao, LEI BAI, Bowen Zhou

Abstract:

The scientific research paradigm is undergoing
a profound transformation owing to the development of Artificial Intelligence (AI). Recent
works demonstrate that various AI-assisted research methods can largely improve research
efficiency by improving data analysis, accelerating computation, and fostering novel idea
generation. To further move towards the ultimate goal (i.e., automatic scientific research),
in this paper, we introduce DOLPHIN, a closedloop LLM-driven framework to enhance the automation level of scientific research. DOLPHIN
first generates novel ideas based on feedback
from previous experiments and relevant papers
ranked by the topic and task attributes. Then,
the generated ideas can be implemented using a code template refined and debugged with
the designed exception-traceback-guided local
code structure. Finally, DOLPHIN automatically analyzes the results of each idea and feeds
the results back to the next round of idea generation. Experiments are conducted on the benchmark datasets of different topics and a subset
of MLE-bench. Results show that DOLPHIN
can continuously improve the performance of
the input topic in a loop. We highlight that
DOLPHIN can automatically propose methods
that are comparable to the state-of-the-art.