Measuring few-shot extrapolation with program induction

Ferran Alet, Javier Lopez-Contreras, Joshua B. Tenenbaum, Tomas Perez, Leslie Pack Kaelbling

Keywords: benchmark, program induction, meta-learning, few-shot learning

TL;DR: A new benchmark for few-shot program induction coming from thousands of small sub-codes obtained by a custom C++ interpreter executes on codes from programming competitions.

Abstract: Neural networks are capable of learning complex functions, but still have problems generalizing from few examples and beyond their training distribution. Meta-learning provides a paradigm to train networks to learn from few examples, but it has been shown that its most popular benchmarks require very limited generalization capabilities. Program induction lies at the opposite end of the spectrum: programs are capable of extrapolating from very few examples, but we still do not know how to efficiently search for complex programs. We propose a common benchmark for both communities, measuring extrapolation from few examples coming from the execution of small programs. These are obtained by leveraging a C++ interpreter on codes from programming competitions; extracting small sub-codes with their corresponding input-output pairs. Statistical analysis and preliminary human experiments show the potential of this benchmark for enabling progress in few-shot extrapolation.