Coding Agent Distillation

XitingLiu, Diankun Wu, Luyu Li

Keywords: coding agent, model distillation

Abstract: Recent advances in large language models (LLMs) have enabled powerful coding agents such as SWE-Agent and OpenDevin that can autonomously plan, write, execute, and debug code. However, these systems rely heavily on massive proprietary models (e.g., GPT-4), making them costly to deploy and unsuitable for private or on-premise use. This research proposes a method to distill the coding-agent capabilities of large LLMs into smaller models (7B--30B parameters) through behavior imitation. By collecting large-model execution trajectories---comprising task instructions, tool invocations, and debugging steps---we train a compact model to reproduce multi-step reasoning and structured code generation. Experiments will be conducted on SWE-Bench and additional self-constructed datasets to evaluate task success rate, tool-use accuracy, and computational efficiency. The proposed approach aims to achieve 70--80\% of large-model performance while significantly reducing inference cost and enabling secure private deployment of intelligent coding assistants. This work contributes to the growing research on agent distillation, offering a lightweight and deployable framework for coding automation.

Submission Number: 16