Agent Skill Acquisition for LLMs via CycleQD

So Kuroki, Taishi Nakamura, Takuya Akiba, Yujin Tang

Keywords: Large Language Models, Skill Acquisition, Quality Diversity

Abstract: Training large language models to acquire specific skills remains a challenging endeavor. Conventional training approaches often struggle with data distribution imbalances and inadequacies in objective functions that do not align well with task-specific performance. To address these challenges, we introduce CycleQD, a novel approach that leverages the Quality Diversity framework through a cyclic adaptation of the algorithm, along with a model merging based crossover and an SVD-based mutation. In CycleQD, each task’s performance metric is alternated as the quality measure while the others serve as the behavioral characteristics. This cyclic focus on individual tasks allows for concentrated effort on one task at a time, eliminating the need for data ratio tuning and simplifying the design of the objective function. Empirical results from AgentBench indicate that applying CycleQD to LLAMA3-8B-INSTRUCT models not only enables them to surpass traditional fine-tuning methods in coding, operating systems, and database tasks, but also achieves performance on par with GPT-3.5-TURBO, which potentially contains much more parameters, across these domains. Furthermore, our method is general and can be applied to image segmentation models, highlighting its applicability across different domains. Our code is available at \url{https://github.com/SakanaAI/CycleQD}.

Submission Number: 22