Take No Shortcuts! Stick to the Rubric: A Method for Building Trustworthy Short Answer Scoring Models

Yuya Asazuma, Hiroaki Funayama, Yuichiroh Matsubayashi, Tomoya Mizumoto, Paul Reisert, Kentaro Inui

Abstract: This paper introduces a new strategy to enhance the trustworthiness of Short Answer Scoring (SAS) systems used in educational settings. Although the development of scoring models with high accuracy has become feasible due to advancements in machine learning methods, particularly recent Transformers, there is a risk of shortcut learning using superficial cues present in training data, leading to behaviors that contradict rubric standards and thus raising issues of model trustworthiness. To address this issue, we introduce an efficient strategy that aligns the features of responses with rubric criteria, mitigating shortcut learning and enhancing model trustworthiness. Our approach includes a detection method that employs a feature attribution method to identify superficial cues and a correction method that re-trains the model to align with annotations related to the rubric, thereby suppressing these superficial cues. Our quantitative experiments demonstrate the effectiveness of our method in consistently suppressing superficial cues, contributing to more trustworthy automated scoring of descriptive questions.