Is one annotation enough? - A data-centric image classification benchmark for noisy and ambiguous label estimationDownload PDF

16 May 2022, 07:22 (modified: 10 Oct 2022, 10:13)NeurIPS 2022 Datasets and Benchmarks Readers: Everyone
Keywords: dataset, benchmark, ambiguity, noisy
TL;DR: A multi-domain data-centric benchmark for investigating the ambiguity and noise of human annotations on deep learning
Abstract: High-quality data is necessary for modern machine learning. However, the acquisition of such data is difficult due to noisy and ambiguous annotations of humans. The aggregation of such annotations to determine the label of an image leads to a lower data quality. We propose a data-centric image classification benchmark with nine real-world datasets and multiple annotations per image to allow researchers to investigate and quantify the impact of such data quality issues. With the benchmark we can study the impact of annotation costs and (semi-)supervised methods on the data quality for image classification by applying a novel methodology to a range of different algorithms and diverse datasets. Our benchmark uses a two-phase approach via a data label improvement method in the first phase and a fixed evaluation model in the second phase. Thereby, we give a measure for the relation between the input labeling effort and the performance of (semi-)supervised algorithms to enable a deeper insight into how labels should be created for effective model training. Across thousands of experiments, we show that one annotation is not enough and that the inclusion of multiple annotations allows for a better approximation of the real underlying class distribution. We identify that hard labels can not capture the ambiguity of the data and this might lead to the common issue of overconfident models. Based on the presented datasets, benchmarked methods, and analysis, we create multiple research opportunities for the future directed at the improvement of label noise estimation approaches, data annotation schemes, realistic (semi-)supervised learning, or more reliable image collection.
Supplementary Material: pdf
Open Credentialized Access: N/A
Dataset Url:
Dataset Embargo: N/A
License: All information about the licenses can be found in the Repository alongside the source code at
Author Statement: Yes
Contribution Process Agreement: Yes
In Person Attendance: Yes
8 Replies