Intervention-Based Alignment of Code Search with Execution Feedback

Hojae Han, Minsoo Kim, seung-won hwang, Nan Duan, Shuai Lu

Submission Type: Regular Long Paper

Submission Track: NLP Applications

Keywords: Code Search, Misalignment, Reinforcement Learning, Intervention

TL;DR: Reinforcement Learning with Code Intervention to Align Model Decision and Execution Feedback in Code Search

Abstract:

One of the fundamental goals in code search is to retrieve a functionally correct code for a given natural language query.
As annotating for correctness requires executing test cases (i.e. obtaining execution feedback), existing code search training datasets approximate text-code co-occurrences as positive execution feedback. However, this approximation may misalign models’ retrieval decisions from ground-truth correctness.
To address such limitation, we propose Code Intervention-based Reinforcement Learning (CIRL) that perturbs training code to result in misalignment (i.e. code intervention), then tests models’ decisions and corrects them with the execution feedback by reinforcement learning. The first technical contribution of CIRL is to induce the execution feedback from perturbation, without actual execution. Secondly, CIRL introduces structural perturbations using abstract syntax trees, going beyond simple lexical changes. Experimental results on various datasets demonstrate the effectiveness of CIRL compared to conventional approaches.

Submission Number: 5228