Using NLP to Quantify Program Decomposition in CS1

Charis Charitsis, Chris Piech, John C. Mitchell

Abstract: Decomposition is a problem-solving technique that is essential to software development. Nonetheless, it is perceived as the most challenging programming skill for learners to master. Researchers have studied decomposition in introductory programming courses through guided experiments, case studies, and surveys. We believe that the rapid advancements in scientific fields such as machine learning and natural language processing (NLP) opened up opportunities for more scalable approaches. We study the relationship between problem-related entities and functional decomposition. We use an automated system to collect 78,500 code snapshots from two CS1 programming assignments of 250 students and then apply NLP techniques to quantify the learner's ability to break down a problem into a series of smaller, more straightforward tasks. We compare different behaviors and evaluate at scale the impact of decomposition on the time it takes to deliver the solution, its complexity, and the student's performance in the assignment and the course exams. Finally, we discuss the implications of our results for teaching and future research.