Toward Trustworthy Neural Program Synthesis
Track: long paper (up to 9 pages)
Keywords: LLM, Code Generation, Trustworthy AI
TL;DR: Proposing a method for giving well-calibrated probabilistic prediction of program correctness
Abstract: We develop an approach to estimate the probability that a program sampled from
a large language model is correct. Given a natural language description of a programming problem, our method samples both candidate programs as well as candidate predicates specifying how the program should behave. This allows learning
a model that forms a well-calibrated probabilistic prediction of program correctness. Our system also infers the which predicates are useful to explain the behavior of the generated code, and humans preferred these in a human study over raw
language model outputs. Our method is simple, easy to implement, and maintains
state of the art generation accuracy results.
Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.
Submission Number: 8
Loading