Data Reliability Scoring

ICLR 2026 Conference Submission13181 Authors

Keywords: Reliability Scoring, Gram Determinant Score, Information Elicitation, Mechanism Design

TL;DR: We propose the Gram determinant score, an experiment-agnostic method for reliability scoring that assesses dataset quality without access to ground truth.

Abstract: How can we assess the reliability of a dataset without access to ground truth? We introduce the problem of reliability scoring for datasets collected from potentially strategic sources. The true data are unobserved, but we see outcomes of an unknown statistical experiment that depends on them. To benchmark reliability, we define ground-truth–based orderings that capture how much reported data deviate from the truth. We then propose the Gram determinant score, which measures the volume spanned by vectors describing the empirical distribution of the observed data and experiment outcomes. We show that this score preserves several ground-truth-based reliability orderings and, uniquely up to scaling, yields the same reliability ranking of datasets regardless of the experiment -- a property we call experiment agnostic. Experiments on synthetic noise models, CIFAR-10 embeddings, and real employment data demonstrate that the Gram determinant score effectively captures data quality across diverse observation processes.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 13181