Modeling and Correcting Bias in Sequential Evaluation

Jingyan Wang, Ashwin Pananjady

Abstract: We consider the problem of sequential evaluation, in which an evaluator observes candidates in a sequence and assigns scores to these candidates in an online, irrevocable fashion. Sequential bias refers to dependencies between the evaluation outcome and the order in which the candidates appear, and extensive empirical studies have established its existence in many applications. Motivated by the psychology literature, we propose a natural model for the evaluator's rating process that captures the lack of calibration inherent to such a task. We conduct crowdsourcing experiments to demonstrate various facets of our model, propose a near-linear time, online algorithm for bias correction, and show that it is near-optimal in two canonical ranking metrics.