S-Agent: an Agent Collaborative Framework \\Inspired by the Scientific Methodology
Abstract: An increasing number of advancements have been accomplished in agents empowered by Large Language Models (LLM), particularly in resolving simple dialogue tasks. However, existing agents still face intractable robustness issues for solving complex tasks, encountering the cascading hallucinations induced by multi-step invocations of LLM. Certain recent studies utilize multi-step reasoning, planning strategies, and domain workflows to improve the success rate of complex tasks, yet they neglect the scientific methodology that encompasses the accumulated wisdom derived from centuries of scientific inquiry. Drawing inspiration from the scientific methodology, we propose the S-Agent - an agent collaborative framework meticulously designed to actively experiment and refine theories based on the analysis of experimental results, thereby enhancing the deductive capabilities of LLMs and complementing their inductive and communicative strengths. Additionally, we introduce an innovative parallel planning methodology, wherein agents with identical roles collaborate to simultaneously address the same inquiry. Extensive experiments demonstrate the effectiveness and efficiency of our approach. Notably, we achieve a new state-of-the-art $33.3\%$ pass@1 accuracy on the LeetcodeHardGym coding benchmark and a relatively good $96.3\%$ pass@1 on HumanEval with GPT-4.
Paper Type: Long
Research Area: NLP Applications
Research Area Keywords: NLP Applications, Code generation
Contribution Types: Model analysis & interpretability, NLP engineering experiment
Languages Studied: English
Submission Number: 2430
Loading