Black-Box Adversarial Attacks on LLM-Based Code Completion

Slobodan Jenko, Jingxuan He, Niels Mündler, Mark Vero, Martin Vechev

Keywords: code completion, security, code security, adversarial attacks, black-box, large language models

TL;DR: We introduce the first black-box adversarial attack on commercial code completion engines for insecure code generation.

Abstract: Modern code completion engines, powered by large language models (LLMs), assist millions of developers with their impressive capabilities to generate functionally correct code. As such it is crucial to investigate their security implications. In this work, we present INSEC, the first black-box adversarial attack designed to manipulate modern LLM-based code completion engines into generating vulnerable code. INSEC works by injecting an attack string as a short comment in the completion input. The attack string is crafted through a query-based optimization procedure starting from a set of initialization schemes. We demonstrate INSEC's broad applicability and effectiveness by evaluating it on various state-of-the-art open-source models and black-box commercial services (e.g., OpenAI API and GitHub Copilot). We show that on a diverse set of security-critical test cases covering 16 CWEs across 5 programming languages, INSEC significantly increases the rate of generated insecure code by ~50%, while upholding the engines' capabilities of producing functionally correct code. Moreover, due to its black-box nature, developing INSEC does not require expensive local compute and costs less than 10 USD by querying remote APIs, thereby enabling the threat of widespread attacks.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 13941