Jigsaw: Large language models meet program synthesis

Naman Jain, Skanda Vaidyanath, Arun Iyer, Nagarajan Natarajan, Suresh Parthasarathy Iyengar, Sriram K. Rajamani, Rahul Sharma

Abstract:

Large pre-trained language models such as GPT-3 [10], Codex [11],
and Google’s language model [7] are now capable of generating
code from natural language specifications of programmer intent.
We view these developments with a mixture of optimism and caution. On the optimistic side, such large language models have the
potential to improve productivity by providing an automated AI
pair programmer for every programmer in the world. On the cautionary side, since these large language models do not understand
program semantics, they offer no guarantees about quality of the
suggested code. In this paper, we present an approach to augment
these large language models with post-processing steps based on
program analysis and synthesis techniques, that understand the
syntax and semantics of programs. Further, we show that such techniques can make use of user feedback and improve with usage. We
present our experiences from building and evaluating such a tool
Jigsaw, targeted at synthesizing code for using Python Pandas API
using multi-modal inputs. Our experience suggests that as these
large language models evolve for synthesizing code from intent,
Jigsaw has an important role to play in improving the accuracy of
the systems.