Cracking the Code: Evaluating Zero-Shot Prompting Methods for Providing Programming Feedback

Niklas Ippisch, Anna-Carolina Haensch, Markus Herklotz, Jan Simson, Jacob Beck, Malte Schierholz

Track: tiny / short paper (up to 5 pages)

Keywords: Prompt Engineering, Feedback Evaluation, R Programming Education

TL;DR: We developed a systematic evaluation framework for assessing LLM responses on programming errors and conducted a case study.

Abstract: We introduce an evaluation framework to assess the feedback given by large language models (LLMs) under different prompt engineering techniques and conduct a case study, systematically varying prompts to examine their influence on feedback quality for common programming errors in R. Our findings suggest that prompts recommending a stepwise approach improve precision, whereas omitting explicit details on which data to analyze can bolster error identification.

Submission Number: 12