Report-Sensitive Spot-checking in Peer Grading Systems

Hedayat Zarkoob, Hu Fu, Kevin Leyton-Brown

Abstract: Peer grading systems make large courses more scalable, provide students with faster and more detailed feedback, and help teach students to think critically about the work of others. Various recent implementations of peer grading mechanisms make such systems relatively easy to deploy in practice [2, 11, 24]. The broader adoption of such systems faces a common, critical obstacle: motivating students to provide accurate grades. A natural solution is asking multiple students to grade the same assignment and rewarding them based on their behavior (e.g., based on the extent to which their grades agree with the grades given by other students). Such solutions have been explored in detail in a large literature on peer prediction, which considers how to incentivize agents to truthfully disclose unverifiable private information [4, 7-10, 12-17, 22, 23]. Unfortunately, almost all known peer prediction mechanisms also give rise to uninformative equilibria in which agents do not reveal their private information; e.g., all students grading an assignment favorably regardless of its quality [1, 8, 10, 17, 22]. Human experiments show that such strategic behavior does arise in practice [5].