Decision-margin consistency: a principled metric for human and machine performance alignment
Track: Proceedings Track
Keywords: error consistency, human-machine alignment, vision
TL;DR: This paper introduces a new metric, decision-margin consistency, to better compare human and machine decision-making by accounting for variability in human responses
Abstract: Understanding the alignment between human and machine perceptual decision-making is a fundamental challenge. While most current vision deep neural networks are deterministic and produce consistent outputs for the same input, human perceptual decisions are notoriously noisy. This noise can originate from perceptual encoding, decision processes, or even attentional fluctuations, leading to different responses for the same stimulus across trials. Thus, any meaningful comparison between human-to-human or human-to-machine decisions must take this internal noise into account to avoid underestimating alignment. In this paper, we introduce the \textbf{decision-margin consistency metric}, which draws on signal detection theory, by incorporating both the variability in decision difficulty across items and the noise in human responses. By focusing on decision-margin distances—-continuous measures of signal strength underlying binary outcomes—-our method can be applied to both model and human systems to capture the nuanced agreement in item-level difficulty. Applying this metric to existing visual categorization datasets reveals a dramatic increase in human-human agreement relative to the standard error consistency metric. Further, human-to-machine agreement showed only a modest increase, highlighting an even larger representational gap between these systems on these challenging perceptual decisions. Broadly, this work underscores the importance of accounting for internal noise when comparing human and machine error patterns, and offers a new principled metric for measuring representational alignment for biological and artificial systems
Submission Number: 47
Loading