Trust-aware Peer Assessment using Multi-armed Bandit Algorithms

Hou Pong Chan, Tong Zhao, Irwin King

Abstract: Massive Open Online Coursers (MOOCs) offer a convenient way for people to access quality courses via the internet. However, the problem of grading open-ended assignments at such a large scale still remains challenging. Although peer assessment have been proposed to handle the large-scale grading problem in MOOCs, existing methods still suffer several limitations: (1) most current peer assessment research ignore the importance of how to allocate the assessment tasks among peers, (2) existing approaches for peer grading learn the complete ranking in an offline manner, (3) theoretical analysis for trust-aware peer grading is missing. In this work, we consider the case that we have prior knowledge about all students' reliability. We formulate the problem of peer assessment as a sequential noisy ranking aggregation problem. We derive a trust-aware allocation scheme for peer assessment to maximize the probability of constructing a correct ranking of assignments with a budget constraint.Moreover, we also derive an upper bound for the probability of prediction error on the inferred ranking of assignments. Furthermore, we propose the Trust-aware Ranking-based Multi-armed Bandit Algorithms to sequentially allocate the assessment tasks to the students based on the derived allocation scheme and learn an accurate peer grading result by taking students' reliability into consideration.