ALGO: Synthesizing Algorithmic Programs with Generated Oracle Verifiers

Kexun Zhang, Danqing Wang, Jingtao Xia, William Yang Wang, Lei Li

Keywords: Large Language Models, Code Generation, Code Intelligence, Automatic Verification

TL;DR: We propose an algorithm synthesis framework that leverages LLM-generated oracles to guide the creation of efficient algorithms and verify their correctness.

Abstract:

Large language models (LLMs) excel at implementing code from functionality descriptions but struggle with algorithmic problems that require not only implementation but also identification of the suitable algorithm. Moreover, LLM-generated programs lack guaranteed correctness and require human verification. To address these challenges, we propose ALGO, a framework that synthesizes Algorithmic programs with LLM-Generated Oracles to guide the generation and verify their correctness. ALGO first generates a reference oracle by prompting an LLM to exhaustively enumerate all the combinations of relevant variables. This oracle is then utilized to guide an arbitrary search strategy in exploring the algorithm space and to verify the synthesized algorithms. Our study shows that the LLM-generated
oracles are correct for 88% of the cases. With the oracles as verifiers, ALGO can be integrated with any existing code generation model in a model-agnostic manner to enhance its performance. Experiments show that when equipped with ALGO, we achieve an 8× better one-submission pass rate over the Codex model and a 2.6× better one-submission pass rate over CodeT, the current state-of-the-art model on CodeContests. We can also get 1.3× better pass rate over the ChatGPT Code Interpreter on unseen problems. The problem set we used for testing, the prompts we used, the verifier and solution programs, and the test cases generated by ALGO
are available at https://github.com/zkx06111/ALGO.

Submission Number: 10278