I Speak, You Verify: Toward Trustworthy Neural Program Synthesis

Darren Key, Wen-Ding Li, Kevin Ellis

Keywords: program synthesis, large language models

TL;DR: Large language models over source code can be made more trustworthy when they jointly generate programs and specifications

Abstract: We develop an approach for improving the trustworthiness and overall accuracy of programs synthesizers based on large language models for source code. Given a natural language description of a programming problem, our method samples both candidate programs as well as candidate predicates specifying what the program should compute. Our method learns to analyze the agreement between programs and predicates to judge both which program is most likely to be correct, and also judge whether the language model is able to solve the programming problem in the first place. This latter capacity allows favoring high precision over broad recall: fostering trust by only proposing a program when the system is certain that it is correct.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Applications (eg, speech processing, computer vision, NLP)