Semi-Supervised Learning via Weight-aware Distillation under Class Distribution Mismatch

Suyun Zhao

Abstract:

Semi-Supervised Learning (SSL) under class distribution mismatch aims to tackle a challenging problem wherein unlabeled data contain lots of unknown categories unseen in the labeled ones. In such mismatch scenarios, traditional SSL suffers severe performance damage due to the harmful invasion of the instances with unknown categories into the target classiffer. In this study, by strict mathematical reasoning, we reveal that the SSL error under class distribution mismatch is composed of pseudo-labeling error and invasion error, both of which jointly bound the SSL population risk. To alleviate the SSL error, we propose a robust SSL framework called Weight-Aware Distillation
(WAD) that, by weights, selectively transfers knowledge beneffcial to the target task from unsupervised contrastive representation to the target classiffer. Speciffcally, WAD captures adaptive weights and high-quality pseudolabels to target instances by exploring point mutual information (PMI) in representation space to maximize the role of unlabeled data and fflter unknown categories. Theoretically, we prove that WAD has a tight upper bound of population risk under class distribution mismatch. Experimentally, extensive results demonstrate that WAD outperforms ffve state-of-the-art SSL approaches and one standard baseline on two benchmark datasets, CIFAR10 and CIFAR100, and an artiffcial cross-dataset.