Discovering Architectures via an Evolutionary Agentic Framework

Weixian Xu, Yixiu Liu, Yang Nan, Lyumanshan Ye, Xiangkun Hu, Zhen Qin, Pengfei Liu

Keywords: LLM; Agent; Linear attention

TL;DR: We created an LLM-powered agent that automates the entire scientific research loop. We pointed it at the linear attention problem, and it autonomously discovered 105 novel architectures that beat current SOTA models.

Abstract: In complex, long-horizon tasks such as scientific discovery, Large Language Models (LLMs) have primarily served as assistants to human researchers rather than acting as autonomous agents capable of driving innovation from hypothesis to discovery. In this paper, we attempt to empower an LLM to not only conduct the entire scientific workflow end-to-end but also to evolve its strategies by learning from experimental outcomes. Our system manages the process from hypothesizing novel ideas and implementing code to conducting experiments and analyzing results. Specifically, we introduce the \modelname framework, which utilizes specialized agents—a Researcher for proposing ideas, an Engineer for evaluation, and an Analyst for interpreting outcomes—to autonomously navigate the research lifecycle. We validated our approach in the challenging domain of linear attention, where our LLM agent conducted 1,773 iterative experiments, leading to the discovery of 105 entirely new architectures. These novel designs outperform existing state-of-the-art (SOTA) models, with their effectiveness confirmed across various model scales and benchmarks. In addition, we conducted a detailed analysis of the LLM's emergent design patterns, providing valuable insights for the research community. We have open-sourced our code and the collection of discovered SOTA models.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 8555