A Label Aggregation Method Using Worker Quality in Crowdsourcing

Nana Ota, Yu Suzuki

Abstract: In this study, we propose a method to improve the quality of labels by considering the quality of workers. Our method constructs models that imitate workers’ work to increase the pseudo-data. We calculate the worker’s agreement rate using the pseudo-data obtained using the worker’s imitation model and data worked by the worker. We calculate the weights of the worker’s vote using the worker’s agreement rate. The labels obtained by applying our method would be better quality than those obtained by a majority vote. Then, we experimented to verify the effectiveness of our method. The accuracy of the labels obtained by applying our method was 0.6685. We can reduce the influence of vandal workers and obtain high-quality labels.

External IDs:dblp:conf/iiwas/OtaS23