On Sampling Top-K Recommendation Evaluation

Open Webpage

Zhi Liu, Dong Li, Ruoming Jin, Jing Gao

Published: 01 Jun 2020, Last Modified: 15 May 2025Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’20)EveryoneCC BY-NC-SA 4.0
Abstract: Recently, Rendle has warned that the use of sampling-based top-$k$ metrics might not suffice, casting doubt on many deep learning and classic recommendation studies. We thoroughly investigate the relationship between sampling top-$k$ (SHR@k) and global top-$K$ Hit-Ratios (HR@K). By formulating a mapping function $f$ so that $SHR@k \approx HR@f(k)$, we demonstrate both theoretically and empirically that sampling top-$k$ provides an accurate approximation of its exact counterpart and consistently identifies the same best models.