How does AI code with repository? An evaluation of ICL strategies for repository code generation

ACL ARR 2025 July Submission1414 Authors

Abstract: Retrieval-Augmented Generation (RAG) is the main method to embed context information Language Model (LM) pipelines. However, repository-aware code generation pose a challenge to off-the-shelf RAG due to lack of specificity of traditional embedders, usually trained to handle context-inespecific coding benchmarks such as HumanEval, MBPP and APPS. In order to create reliable pipelines, without relying on any retriever or generator finetuning, we studied the impact of different contexts: 1) We firstly include the local scope of the retrieved functions and methods; 2) we extend it to include the whole function file in the context; 3) we evaluate the impact of the implementation in the same file of the new function (``Infile'' context); 4) we combine the entire retrieved function file with Infile; finally (5) we evaluate the ability of Language Models (LMs) to self-generate documentation and use them to implement new repository functions. Our experiments show the necessity of keeping the whole current, and retrieved file in the context as opposed to specific methods and classes. With this setup, we reach the state-of-the-art performance in CoderEval benchmark employing the open-source small-scale Llama3.1-8B-Instruct without finetuning the generator or the retriever, and without relying on compiler feedback.

Paper Type: Long

Research Area: NLP Applications

Research Area Keywords: code generation and understanding, code models, retrieval-augmented generation

Contribution Types: NLP engineering experiment

Languages Studied: En, Python and Java (code only)

Reassignment Request Area Chair: This is not a resubmission

Reassignment Request Reviewers: This is not a resubmission

A1 Limitations Section: This paper has a limitations section.

A2 Potential Risks: N/A

B Use Or Create Scientific Artifacts: Yes

B1 Cite Creators Of Artifacts: Yes

B1 Elaboration: All datasets and models papers and/or reports have been properly cited in the paper.

B2 Discuss The License For Artifacts: Yes

B2 Elaboration: All models and datasets used are publicly available and allowed for research purposes.

B3 Artifact Use Consistent With Intended Use: Yes

B3 Elaboration: The whole paper is a research analysis about the impact of context formation for repository-aware code generation. All models and dataset are publicly available and allowed for research purposes.

B4 Data Contains Personally Identifying Info Or Offensive Content: N/A

B5 Documentation Of Artifacts: Yes

B5 Elaboration: In the paper we briefly describe the CoderEval dataset, that has been already used in other researchs and it is publicly available.

B6 Statistics For Data: Yes

B6 Elaboration: We employed the CoderEval dataset for our experiments in its standard format. In Section 4, we clearly say the number of examples of the dataset, the evaluation setups and the metrics employed.

C Computational Experiments: Yes

C1 Model Size And Budget: Yes

C1 Elaboration: We report the number of parameters for each model and the GPUs specs.

C2 Experimental Setup And Hyperparameters: Yes

C2 Elaboration: There is an ablation study section evaluating different setups of our proposed solution.

C3 Descriptive Statistics: Yes

C3 Elaboration: The only metric employed in the paper is the pass@1 (chance of a successful code generation at first try) which typically does not have error bars and other statistics reported in prior works.

C4 Parameters For Packages: Yes

C4 Elaboration: All parameters are clearly stated in the paper.

D Human Subjects Including Annotators: No

D1 Instructions Given To Participants: N/A

D2 Recruitment And Payment: N/A

D3 Data Consent: N/A

D4 Ethics Review Board Approval: N/A

D5 Characteristics Of Annotators: N/A

E Ai Assistants In Research Or Writing: No

E1 Information About Use Of Ai Assistants: N/A

Author Submission Checklist: yes

Submission Number: 1414