"Can we reach agreement?": A context- and semantic-based clustering approach with semi-supervised text-feature extraction for finding disagreement in peer-assessment formative feedback.

M Parvez Rashid, Divyang Doshi

Abstract: In the process of review for assessing a piece of work, agree- ment or consensus among reviewers is vital to review qual- ity. As classroom peer assessments are undertaken by na ̈ıve peers, disagreement among peer assessors can confuse the assessees and lead them to question the review process. Al- though there are methods like inter-rater reliability (IRR) to measure disagreement in summative feedback, in the au- thors’ knowledge, there is no method for finding disagree- ments within formative feedback. It may take more time and effort for the instructor to review the feedback to find dis- agreements than it would to simply perform an expert review without involving peer assessors. An automated method can help locate disagreements among reviewers. In this work, we used a clustering algorithm and NLP techniques to find dis- agreement in formative feedback. As the review comments are related by context and semantics, we implemented a semi-supervised approach to fine-tune the SentenceTrans- former model to capture the context and semantics-based relation among the review texts, which in turn improved the comment clustering performance.