Who and How: Using Sentence-Level NLP to Evaluate Idea Completeness

Martin Ruskov

Abstract: Real-time feedback is very important, yet challenging to provide for free-text learner contributions in Technology-Enhanced Learning. We study whether a generic NLP pipeline can identify completeness features of learner ideas during security training. We apply PoS Tagging and Dependency Parsing on contextualised short texts, collected within a dedicated learning environment and we compare the results to an expert-annotated ground truth. We scan these contributions for the absence of responsible stakeholder (who) or featured action (how). A total of 1174 contributions in two security domains were analysed. We report precision on who ( $$PPV=0.929$$ ) and on how ( $$PPV=0.691$$ ). We consider the first result to be sufficient to provide real-time formative feedback for the case of absent who. Our results suggest that for the purposes of providing feedback in free input problem-solving exercises, generic transformer pipelines without fine-tuning can achieve good performance on stakeholder identification.