Score-Based Density Estimation from Pairwise Comparisons

ICLR 2026 Conference Submission15889 Authors

Keywords: score-based methods, pairwise comparisons, density estimation, elicitation, random utility models, tempering

TL;DR: We show how to estimate densities solely from pairwise comparisons. We establish a relationship between the target density and tempered density of the preferred choices, and provide a score-based method that recovers the target density.

Abstract: We study density estimation from pairwise comparisons, motivated by expert knowledge elicitation and learning from human feedback. We relate the unobserved target density to a tempered winner density (marginal density of preferred choices), learning the winner's score via score-matching. This allows estimating the target by `de-tempering' the estimated winner density's score. We prove that the score vectors of the belief and the winner density are collinear, linked by a position-dependent tempering field. We give analytical formulas for this field and propose an estimator for it under the Bradley-Terry model. Using a diffusion model trained on tempered samples generated via score-scaled annealed Langevin dynamics, we can learn complex multivariate belief densities of simulated experts, from only hundreds to thousands of pairwise comparisons.

Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Submission Number: 15889