Is a picture of a bird a bird? A mixed-methods approach to understanding diverse human perspectives and ambiguity in machine vision models
Keywords: Disagreements, Ambiguity, Machine vision
TL;DR: Humans' personal experiences and perspectives introduce subjectivity into image-labeling tasks for machine vision models due to factors like ambiguous image content, varied rater backgrounds, and how the labeling task itself is set up.
Abstract: Human experiences are complex and subjective. This subjectivity is reflected in the way people label images for machine vision models. While annotation tasks are often assumed to deliver objective results, this assumption does not allow for the subjectivity of human experience. This paper examines the implications of subjective human judgments in the behavioral task of labeling images used to train machine vision models. We identify three primary sources of ambiguity: (1) depictions of labels in the images can be simply ambiguous, (2) raters' backgrounds and experiences can influence their judgments and (3) the way the labeling task is defined can also influence raters' judgments. By taking steps to address these sources of ambiguity, we can create more robust and reliable machine vision models.
Primary Subject Area: Data collection and benchmarking techniques
Paper Type: Research paper: up to 8 pages
Participation Mode: In-person
Confirmation: I have read and agree with the workshop's policy on behalf of myself and my co-authors.
Submission Number: 59
Loading