Robust annotation aggregation in crowdsourcing via enhanced worker ability modeling

Ju Chen, Jun Feng, Shenyu Zhang, Xiaodong Li, Hamza Djigal

Abstract: Highlights•Modeling a worker’s ability as a confusion matrix works better than a single value.•Imbalanced classes and inadequate data are issues of confusion matrices estimation.•Gaussian distribution and softmax are combined to improve worker ability estimation.•A computationally efficient algorithm is devised to approximate the posterior.•Experiments on 24 datasets verify the effectiveness and efficiency of our model.