Differential evolution-based weighted soft majority voting for crowdsourcing

Fangna Tao, Liangxiao Jiang, Chaoqun Li

Abstract: Crowdsourcing has attracted considerable attention in recent years. A large amount of labeled data can be obtained efficiently and cheaply from the crowdsourcing platform. Obviously, the labeling quality of crowd workers directly influences the quality of the labeled data. Although a small amount of label integration strategies have recently noticed the differences in the quality of crowd workers labeling different instances, which just utilize the statistical characteristics of multiple noisy labels to estimate the quality of crowd workers and thus are rough and sub-optimal. In addition, they can only deal with binary classification problems, which restricts the practical applications of crowdsourcing. To simultaneously solve these two issues, we propose three differential evolution-based weighted soft majority voting strategies for multi-class classification. In our proposed strategies, we exploit a differential evolution (DE) algorithm to estimate the quality of crowd workers labeling different instances by minimizing the Error, Gini and Entropy of weighted multiple noisy labels. Extensive experimental results on simulated and real-world datasets show that our proposed strategies significantly outperform all the other existing state-of-the-art label integration strategies.