Solving probability and statistics problems by probabilistic program synthesis at human level and predicting solvability

Leonard Tang, Elizabeth Ke, Nikhil Singh, Bo Feng, Derek Austin, Nakul Verma, Iddo Drori

Abstract: We use probabilistic program synthesis to solve questions in MIT and Harvard Probability and Statistics courses. Traditional approaches using the latest GPT-3 language model without program synthesis achieve a solve rate of 0.2 in these classes. In contrast, by turning course questions into probabilistic programs using the latest program synthesis Transformer, OpenAI Codex, and executing the programs, our solve rates are 0.9 and 0.88, which are on par with human performance.