Retrieving Key Topical Sentences with Topic-Aware BERT When Conducting Automated Essay Scoring

Yongchao Wu, Aron Henriksson, Jalal Nouri, Martin Duneld, Xiu Li

Abstract: Automated Essay Scoring (AES) automatically assign scores to essays at scale and may help to support teachers’ grading activities. Recently, AES methods based on deep neural networks (DNN) have significantly improved upon the state-of-the-art performance by learning relations between holistic essay scores and student essays. However, DNN-based AES methods function like black-box, negatively affecting the ability to provide automated writing evaluation (AWE). In this work, we proposed a new method, topic-aware BERT, based on fine-tuning the pre-trained language model to learn relations between essay scores and text representations of student essays as well as topical information in essay writing instructions. Moreover, we propose an approach to automatically retrieve key topical sentences in student essays by probing self-attention maps in intermediate layers of topic-aware BERT. We evaluate the performance of topic-aware BERT to (i) perform AES and (ii) retrieve key topical sentences using the open dataset Automated Student Assessment Prize and a manually annotated dataset, respectively. Our model achieves a strong AES performance compared with previous state-of-the-art DNN-based methods and shows effectiveness in identifying key topical sentences in argumentative essays.