Evaluating Crowdsourcing Participants in the Absence of Ground-Truth

Ramanathan Subramanian, Rómer Rosales, Glenn Fung, Jennifer G. Dy

Abstract: Given a supervised/semi-supervised learning scenario where multiple annotators are available, we consider the problem of identification of adversarial or unreliable annotators.