Automated Assessment of Comprehension Strategies from Self-explanations Using Transformers and Multi-task Learning

Bogdan Nicula, Marilena Panaite, Tracy Arner, Renu Balyan, Mihai Dascalu, Danielle S. McNamara

Abstract: Self-explanation practice is an effective method to support students in better understanding complex texts. This study focuses on automatically assessing the comprehension strategies employed by readers while understanding STEM texts. Data from 3 datasets (N = 11,833) with self-explanations annotated on different comprehension strategies (i.e., bridging, elaboration, and paraphrasing) and an overall quality score was used to train various machine learning models in both single-task and multi-task setups. Our end-to-end neural architecture considers RoBERTa as an encoder applied to the target and self-explanation texts, combined with handcrafted features for assessing text cohesion and filtering out low-quality examples. The best configuration obtained a .699 weighted F1-score for the overall self-explanation quality.