An Empirical Study on the Effect of Quick and Careful Labeling Styles in Image Annotation

Chia-Ming Chang, Xi Yang, Takeo Igarashi

Keywords: Cognitive Psychology, Labeling Style, Non-Expert Data Annotation, Data Collection, Machine Learning

Abstract: Assigning a label to difficult data requires a long time, particularly when non-expert annotators attempt to select the best possible label. However, there have been no detailed studies exploring a label selection style during annotation. This is very important and may affect the efficiency and quality of annotation. In this study, we explored the effects of labeling style on data annotation and machine learning. We conducted an empirical study comparing “quick labeling” and “careful labeling” styles in image-labeling tasks with three levels of difficulty. Additionally, we performed a machine learning experiment using labeled images from the two labeling styles. The results indicated that quick and careful labeling styles have both advantages and disadvantages in terms of annotation efficiency, label quality, and machine learning performance. Specifically, careful labeling improves label accuracy when the task is moderately difficult, whereas it is time-consuming when the task is easy or extremely difficult.