Global-Aware Model-Free Self-distillation for Recommendation System

Ang Li, Jian Hu, Wei Lu, Ke Ding, Xiaolu Zhang, Jun Zhou, Yong He, Liang Zhang, Lihong Gu

Abstract: The recommendation performance in our Alipay advertising system may suffer from label noise in training data. Earlier approaches that relied on soft targets typically neglected global similarities. Here, we introduce a novel algorithm called Global-aware Model-free Self-Distillation (GMSD) to create soft targets using the information at the global scale. Specifically, we propose calculating the similarities between the target sample and cluster centers produced by clustering the training dataset. The direct calculation of global similarities requires computation across the full dataset, which is prohibitively expensive. Additionally, we develop a contrastive cluster loss (CCLoss) for limiting the distance between the data of the intra-class to be lower than that of the inter-class to promote samples to be drawn to accurate cluster clusters. Extensive experiments on public and industry datasets demonstrate that GMSD outperforms state-of-the-art self-distillation methods in efficiency and effectiveness.