Generating Hints for Programming Problems Without a Solution

Victor Suciu, Isabel Giang, Bill Zhao, Jessica Runandy, Mitchell Dang

Abstract: We propose a novel way of implementing an Intelligent Tutoring System (ITS) to accompany coding problems without accessible solutions. Previous ITS are good at generating feedback for coding problems with known solutions, but there exists a wide variety of coding platforms that do not publish solutions for many of their problems. These platforms are hugely popular with CS students, so a hint generator that does not require a known solution would benefit a large population of students. We attempted this task using Natural Language Processing (NLP) techniques, specifically BERT feature extraction, to predict solution components. We selected three features that were easy to programmatically detect in solutions - lists, stacks, and queues - and labeled each statement with three booleans denoting whether their solutions contained each feature. Next, we fed the statements through BERT and fit several binary classifiers to the 12th layer to predict which statements had which feature. We were unsuccessful at predicting these features from the problem statements beyond the negative class proportion, possibly due to class imbalance or insufficient BERT pre-training. Because this is a new problem for ITS, we also propose a general benchmark to measure the efficacy of solution-less hint generators in hopes of informing future research.