PatchSynth: a Patch-Text Pre-trained Model
Keywords: PatchSynth, Patch-Text Pretraining, Commit message generation, triple-loss joint learning
TL;DR: PatchSynth is a novel pre-trained Patch-Text model and its novelty is in joint learning for patch and description. It achieves higher performance the state of the art on the task of patch description generation
Abstract: In recent years, patch representation learning has emerged as a necessary research direction for exploiting the capabilities of machine learning in software generation. These representations have driven significant performance enhancements across a variety of tasks involving code changes. While the progress is undeniable, a common limitation among existing models is their specialization: they predominantly excel in either predictive tasks, such as security patch classification, or in generative tasks such as patch description generation. This dichotomy is further exacerbated by a prevalent dependency on potentially noisy data sources. Specifically, many models utilize patches integrated with Abstract Syntax Trees (AST) that, unfortunately, may contain parsing inaccuracies, thus acting as a suboptimal source of supervision. In response to these challenges, we introduce PATCHSYNTH, a novel pre-training framework for patches and natural language text. PATCHSYNTH deploys a triple-loss training strategy for (1) patch-description contrastive learning, which enables to separate patches and descriptions in the embedding space, (2) patch-description matching, which ensures that each patch is associated to its description in the embedding space, and (3) patch-description generation, which ensures that the patch embedding is effective for generation. These losses are implemented for joint learning to achieve good performance in both predictive and generative tasks involving patches. Empirical evaluations focusing on patch description generation, demonstrate that PATCHSYNTH sets new state of the art performance, consistently outperforming the state-of-the-art in metrics like BLEU, ROUGE-L, METEOR, and Recall.
Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 3639
Loading