RETROcode: Leveraging a Code Database for Improved Natural Language to Code Generation

Anonymous

TL;DR: RETRO adaptation for seq2seq in the code generation field

Abstract:

As the availability of text and code has increased, large-scale pre-trained models have demonstrated considerable potential in tackling code generation problems. These models usually apply a supervised fine-tuning approach, training on pairs of natural language problem statements and corresponding ground-truth programs. However, the strategy of increasing the model size and training data quantity, despite potential performance improvements, also inflates computational costs and can lead to overfitting \cite{DS-1000}.
Considering these issues, we introduce RETROcode, a novel adaptation of the RETRO architecture \cite{RETRO} for sequence-to-sequence models, that strategically employs a sizable code database as an auxiliary method for model scaling. Unlike approaches that solely increase model and data size, RETROcode enables the model to directly access a large code database for making predictions. This provides an efficient mechanism to augment language models with substantial-scale memory.
Our work includes an empirical analysis of methods for integrating information from natural language and code from database in the generation process. Leveraging a large database, we outperform classic architectures with similar number of parameters on our test sets and we achieve results that are getting closer to Codex despite it having a significantly larger parameter and training data size.

Paper Type: long

Research Area: Machine Translation

Contribution Types: Approaches to low-resource settings, Approaches low compute settings-efficiency

Languages Studied: English , Python