ADAPTIVE GENERATION OF PROGRAMMING PUZZLES

Ashwin Kalyan, Oleksandr Polozov, Adam Tauman Kalai

TL;DR: We introduce Programming Puzzles, an expressive set of reasoning problems that require minimal understanding of other domains like language. We then introduce TroubleMakers, an algorithm to generate hard programming puzzles.

Abstract:

AI today is far from being able to write complex programs. What type of problems
would be best for computers to learn to program, and how should such problems
be generated? To answer the first question, we suggest programming puzzles as a
domain for teaching computers programming. A programming puzzle consists of a
short program for a Boolean function f(x) and the goal is, given the source code, to
find an input that makes f return True. Puzzles are objective in that one can easily
test the correctness of a given solution x by seeing whether it satisfies f, unlike the
most common representations for program synthesis: given input-output pairs or an
English problem description, the correctness of a given solution is not determined
and is debatable. To address the second question of automatic puzzle generation,
we suggest a GAN-like generation algorithm called “Troublemaker” which can
generate puzzles targeted at any given puzzle-solver. The main innovation is that it
adapts to one or more given puzzle-solvers: rather than generating a single dataset
of puzzles, Tro

Keywords: program synthesis, reasoning, math problems