Weighted Adversarial Learning From Crowds

Ziqi Chen, Liangxiao Jiang, Wenjun Zhang, Chaoqun Li

Abstract: Crowdsourcing services provide a fast and cheap way to annotate instances by employing crowd workers on the Internet. As a result, many learning from crowds (LFC) methods have been proposed in recent years. However, almost all these methods assume that all instances are benign, which makes them vulnerable to adversarial attacks. To improve the model's robustness to adversarial attacks, adversarial LFC (A-LFC) has attracted remarkable attention. A-LFC iteratively updates the true labels’ estimations and the trained model by adversarial learning and uses the model's predictions in turn to help estimate the true labels. In A-LFC, to the best of our knowledge, the true labels’ estimations and the model's predictions are inaccurate and the stopping condition for iterations is rough, which limit the model's performance. To further improve A-LFC, this article proposes weighted A-LFC (WA-LFC). To reduce the impact of misinformation in the true labels’ estimations and model's predictions, our method weights instances for adversarial learning and weights the model's predictions for estimating the true labels. Our method iteratively updates these weights and uses instance-weighted cross-entropy loss to decide when the iterative process should be stopped. Experiments on three real-world datasets show that our method substantially improves the trained model's performance. On average, the test accuracy of our method outperforms that of A-LFC by 10.06% and 27.55% in white-box and black-box attack settings, respectively.