Revealing Student Understanding By Using ChatGPT in Computer Science Assignments

Lorraine Jacques

Abstract:

In recent months, there has been much concern about students using ChatGPT and similar tools to
do their assignments, but what if we as instructors incorporate these tools deliberately? What can the resulting
student work tell us about their understanding and misconceptions? Two programming assignments using
ChatGPT were given to students in a CS 101 course; the resulting student work revealed insights into student
learning in ways that are difficult to capture in traditional programming assignments. A qualitative analysis of
this data suggests the following about student learning: (1) Students prefer to represent code using a
flowchart but have more errors in their representation than those who choose pseudocode, (2) Students are
less likely to error-check the input when modifying existing code than they when incorporating a pregenerated
subroutine into a program they started themselves, and (3) Creating a different approach to an AIgenerated
solution reveals misconceptions student have concerning topics covered much earlier in the course.