Grading Chinese Answers on Specialty Subjective Questions

Dongjin Li, Tianyuan Liu, Wei Pan, Xiaoyue Liu, Yuqing Sun, Feng Yuan

Abstract: It is an important task to grade answers on specialty subjective questions, which is helpful for the supervision of human review and improving the efficiency and quality of review process. Since this grading process should be performed at the same time with human review, there are only a few samples available for each question that can be provided by specialty experts before review process. We investigate the problem of grading Chinese answers on specialty subjective questions with a reference answer in this paper by proposing a grading model that combines two Bi-LSTM networks with attention mechanism. The first part is a sequence to sequence Bi-LSTM network that adopts the pre-trained word embeddings as input. Since there is no embedding for some specialty words, we instead use the fine-grained word embeddings. After the max-pooling on each sentence, we adopt the mutual attention mechanism to learn the matching degree on specialty knowledge between each pair of sentences of answer and reference. Then we adopt another Bi-LSTM with max-pooling to have an overall vector. By concatenating these two vectors from answer and reference, a multilayer perceptron is adopted to predicate the scores. We adopt the real datasets on a national specialty examination to thoroughly verify the model performance against different amount of training data, network structures, pooling strategies and attention mechanisms. The experimental results show the effectiveness of our method.