On Estimating Recommendation Evaluation Metrics under Sampling

Ruoming Jin, Dong Li, Jing Gao, Zhi Liu, Benjamin Mudrak

Abstract: Following critiques of sampling-based top-$k$ metrics, we study how to estimate global evaluation metrics via sampling. We introduce a new research problem of learning the empirical rank distribution and propose two approaches—MLE-based and maximal-entropy estimation—to recover rank distributions from sampled data. Experiments show our estimators accurately approximate global top-$k$ metrics, providing a principled foundation for sampling evaluation.