Who Said What: Modeling individual labelers improves classification

Melody Y. Guan, Varun Gulshan, Andrew M. Dai, Geoffrey E. Hinton

Abstract: Data are often labeled by many different experts, with each expert labeling a small fraction of the data and each sample receiving multiple labels. When experts disagree, the standard approaches are to treat the majority opinion as the truth or to model the truth as a distribution, but these do not make any use of potentially valuable information about which expert produced which label. We propose modeling the experts individually and then learning averaging weights for combining them, possibly in sample-specific ways. This allows us to give more weight to more reliable experts and take advantage of the unique strengths of individual experts at classifying certain types of data. We show that our approach performs better than three competing methods in computer-aided diagnosis of diabetic retinopathy.

Conflicts: google.com

Keywords: Computer vision, Deep learning, Supervised Learning