Code snippets insertion approach for code completion task

Nina Kamkia, Dmitry Stanishevskiy, Alena Fenogenova, Vitaly Kadylin, Denis Kokosinskii, Dmitry Zmitrovich

Keywords: Code completion, LLMs, Benchmark, RAG, Code snippets, GitHub files, Exact match metric

Abstract: The purpose of the code completion task is to continue the following line of the source code. This feature helps developers to write code faster and with fewer errors by providing suggestions for completing the current line of code based on the context and the available libraries and functions. One common but resource-expensive approach to improve model performance on the code completion task is to fine-tune Large Language Models (LLMs). This paper introduces a technique that involves feeding the additional input with ranked portions of code to the LLMs without additional fine-tuning. The proposed approach aims to replicate the development process of programmers by scanning project files for the required code snippets, making the process intuitive and efficient. The paper also discusses the lack of metrics for the task and puts forward a novel metric ClickScore, as well as a new code benchmark RealCode for the code completion task. The paper compares the insertion technique of code snippets with the existing state-of-the-art methods using standard and proposed metrics and demonstrates the approach's effectiveness.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 10119