Go to ACL ARR 2025 February homepageEnhancing Large Language Model’s Capabilities in Open Domains via Autonomous Tool Integration from GitHub
Large Language Models (LLMs) excel in traditional natural language processing tasks but struggle with problems that require complex domain-specific calculations or simulations. While equipping LLMs with external tools to build LLM-based agents can enhance their capabilities, existing approaches lack the flexibility to address diverse and ever-evolving user queries in open domains. Currently, there is also no existing dataset that evaluates LLMs on open-domain knowledge that requires tools to solve.
Notably, the largest open-domain platform is GitHub.
To this end, we introduce \textbf{OpenAct} based on human expert consult and repositories in GitHub. It comprises 339 questions spanning 7 diverse domains that need to be solved with domain-specific methods. In our experiments, even state-of-the-art LLMs and LLM-based agents demonstrate shallow success rates on OpenAct, underscoring the need for a novel approach.
Based on the characteristics of this task, we present \textbf{OpenAgent}, a novel LLM-based Agent system that can tackle evolving queries in open domains through autonomously integrating specialized tools from GitHub. \model employs 1) a hierarchical framework where specialized agents handle specific tasks and can assign tasks to inferior agents, 2) a bi-level experience learning mechanism to learn from both humans' and its own experiences to tackle tool flaws. Experiments demonstrate \model's superior effectiveness and efficiency that significantly outperforms current methods.