ExecRepoBench: Multi-level Executable Code Completion Evaluation

ACL ARR 2025 May Submission6602 Authors

Abstract: Code completion has become an essential tool for daily software development. Existing evaluation benchmarks often employ static methods that do not fully capture the dynamic nature of real-world coding environments and face significant challenges, including limited context length, reliance on superficial evaluation metrics, and potential overfitting to training datasets. In this work, we introduce a novel framework for enhancing code completion in software development through the creation of a repository-level benchmark ExecRepoBench and the instruction corpora RepoInstruct, aim at improving the functionality of open-source large language models (LLMs) in real-world coding scenarios that involve complex interdependencies across multiple files. ExecRepoBench include 1.2K samples from active Python repositories. Plus, we present a multi-level grammar-based completion methodology conditioned on the abstract syntax tree to mask code fragments at various logical units (e.g. statements, expressions, and functions). Then, we fine-tune the open-source LLM with 7B parameters on RepoInstruct to produce a strong code completion baseline model Qwen2.5-Coder-Instruct-C based on the open-source model. Qwen2.5-Coder-Instruct-C is evaluated on ExecRepoBench, which gets both competitive results on code generation and code completion. The deployment of Qwen2.5-Coder-Instruct-C can be used as a high-performance, local service for programming development

Paper Type: Long

Research Area: Generation

Research Area Keywords: Code Large Language Model, Code Completion, parsed abstract syntax tree (AST)

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Data resources, Data analysis

Languages Studied: English

Submission Number: 6602