Code diffusion models are continuous human noise operators

ICLR 2025 Conference Submission12668 Authors

Keywords: Program Repair, Text Diffusion, Code Diffusion, Language Models

TL;DR: Diffusion models act as latent code repair agents and human error generators, enabling precise last-mile repairs and large-scale generation of synthetic data for code repair.

Abstract:

Diffusion for code generates code by iteratively removing noise from the latent representation of a code snippet.
During later steps of the diffusion process, when the code snippet has almost converged, these edits resemble last-mile repairs applied to broken or incomplete code. We evaluate the extent to which these errors are similar to those that humans are faced with and the capability of these models to perform last-mile repair. Our insight has two applications with significant impact for code repair. First, we can leverage the diffusion model for last-mile repair by adding noise to a broken code snippet and resuming the diffusion process. Second, we can leverage the diffusion model to generate an arbitrary amount of training data for other last-mile repair approaches (that are computationally more efficient) by sampling an intermediate program (input) and the final program (output) from the diffusion process. We perform experiments to evaluate both applications, as well as analyze trends in the evolution of representation through the diffusion pipeline providing insights on the reasoning observed.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 12668