Detecting Struggling Students from Interactive Ebook Data: A Case Study Using CSAwesome

Barbara J. Ericson, Hisamitsu Maeda, Paramveer S. Dhillon

Abstract: Ebooks on the Runestone platform contain instructional material (text, images, videos, and a code visualizer/stepper) and a variety of practice problem types (write code problems with unit tests, multiple-choice questions, mixed-up code problems, etc.). User interaction is timestamped and logged. This paper reports on analyses comparing student interaction data to midterm scores for CSAwesome: a College Board endorsed ebook for the Advanced Placement Computer Science A course. We also analyzed mixed-up code (Parsons) problem data in-depth since these are a newer type of practice. Our analysis found that the percent correct on the midterm was most negatively correlated with being in a larger class and most positively correlated with the percent correct on other multiple-choice questions. It was also positively correlated with several other activities including the percent correct on Parsons problems, active code, and the pretest. Interestingly, it was positively correlated with the number of videos viewed, but negatively correlated with the number of videos completed. Next, our analysis of adaptive mixed-up code (Parsons) problems, where the student can ask for help when stuck, found a positive correlation with the number of steps a user completed before asking for help and a negative correlation with the elapsed time before getting help. Looking closely at two Parsons problems, we found that solving each problem more efficiently, i.e., with fewer extra steps, correlated with higher midterm scores. This work could help instructors identify and support struggling students early in a semester and informs the redesign of the instructor dashboard.