A Self-boosted Framework for Calibrated Ranking

Shunyu Zhang, Hu Liu, Wentian Bao, Enyun Yu, Yang Song

Abstract: Scale-calibrated ranking systems are ubiquitous in real-world applications nowadays, which pursue accurate ranking quality and calibrated probabilistic predictions simultaneously. For instance, in the advertising ranking system, the predicted click-through rate (CTR) is utilized for ranking and required to be calibrated for the downstream cost-per-click ads bidding. Recently, multi-objective based methods have been wildly adopted as a standard approach for Calibrated Ranking, which incorporates the combination of two loss functions: a pointwise loss that focuses on calibrated absolute values and a ranking loss that emphasizes relative orderings. However, when applied to industrial online applications, existing multi-objective CR approaches still suffer from two crucial limitations First, previous methods need to aggregate the full candidate list within a single mini-batch to compute the ranking loss. Such aggregation strategy violates extensive data shuffling which has long been proven beneficial for preventing overfitting, and thus degrades the training effectiveness. Second, existing multi-objective methods apply the two inherently conflicting loss functions on a single probabilistic prediction, which results in a sub-optimal trade-off between calibration and ranking.To tackle the two limitations, we propose a Self-Boosted framework for Calibrated Ranking (SBCR). In SBCR, the predicted ranking scores by the online deployed model are dumped into context features. With these additional context features, each single item can perceive the overall distribution of scores in the whole ranking list, so that the ranking loss can be constructed without the need for sample aggregation. As the deployed model is a few versions older than the training model, the dumped predictions reveal what was failed to learn and keep boosting the model to correct previously mis-predicted items. Moreover, a calibration module is introduced to decouple the point loss and ranking loss. The two losses are applied before and after the calibration module separately, which elegantly addresses the sub-optimal trade-off problem. We conduct comprehensive experiments on industrial scale datasets and online A/B tests, demonstrating that SBCR can achieve advanced performance on both calibration and ranking. Our method has been deployed on the video search system of Kuaishou, and results in significant performance improvements on CTR and the total amount of time users spend on Kuaishou.