StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Code

Hannah McLean Babe, Sydney Nguyen, Yangtian Zi, Molly Q Feldman, Carolyn Jane Anderson, Arjun Guha

Keywords: code generation, large language models, prompt engineering

TL;DR: We build the first dataset of Code LLM prompts authored by student programmers, and evaluate several LLMs

Abstract:

Code LLMs are being rapidly deployed and there is evidence that they can make professional programmers more productive. Current benchmarks for code generation measure whether models generate correct programs given an expert prompt. We present the first
benchmark with prompts written by a uniform population of non-experts: students with just one semester of Python programming. StudentEval has 1,749 prompts for 48 problems, written by 80 students. Our students wrote these prompts while working interactively with a Code LLM, and we observed very mixed success rates. We use StudentEval to evaluate contemporary Code LLMs and find that StudentEval is a better discriminator of model performance than existing benchmarks. We analyze the prompts and find significant variation in students’ prompting techniques. We also find that nondeterministic LLM sampling could mislead students into thinking that their prompts are more (or less) effective than they actually are, which has implications for how to teach with Code LLMs.

Primary Area: datasets and benchmarks

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4866