Crowd Teaches the Machine: Reducing Cost of Crowd-Based Training of Machine Classifiers

Radoslaw Nielek, Filip Georgiew, Adam Wierzbicki

Abstract: Crowdsourcing platforms are very frequently used for collecting training data. Quality assurance is the most obvious problem but not the only one. This work proposes iterative approach which helps to reduce costs of building training/testing datasets. Information about classifier confidence is used for making decision whether new labels from crowdsourcing platform are required for this particular object. Conducted experiments have confirmed that proposed method reduces costs by over 50 % in best scenarios and at the same time increases the percentage of correctly classified objects.