Spot Check Equivalence: an Interpretable Metric for Information Elicitation Mechanisms

Shengwei Xu, Yichi Zhang, Paul Resnick, Grant Schoenebeck

Keywords: Algorithmic Game Theory, Information Elicitation, Incentive for Effort, Peer Prediction

Abstract: Because high-quality data is like oxygen for AI systems, effectively eliciting information from crowdsourcing workers has become a first-order problem for developing high-performance machine learning algorithms. Two prevalent paradigms, spot-checking and peer prediction, enable the design of mechanisms to evaluate and incentivize high-quality data from human labelers. So far, at least three metrics have been proposed to compare the performances of these techniques [Zhang and Schoenebeck 2023, Gao et al. 2016, Burrell and Schoenebeck 2023]. However, different metrics lead to divergent and even contradictory results in various contexts. In this paper, we harmonize these divergent stories, showing that two of these metrics are actually the same within certain contexts and explain the divergence of the third. Moreover, we unify these different contexts by introducing Spot Check Equivalence, which offers an interpretable metric for the effectiveness of a peer prediction mechanism. Finally, we present two approaches to compute spot check equivalence in various contexts, where simulation results prove the effectiveness of our proposed metric.

Track: COI (submissions co-authored by SAC)

Submission Guidelines Scope: Yes

Submission Guidelines Blind: Yes

Submission Guidelines Format: Yes

Submission Guidelines Limit: Yes

Submission Guidelines Authorship: Yes

Student Author: Yes

Submission Number: 2200