Enhancing Semi-supervised Learning with Zero-shot Pseudolabels

Jichan Chung, Irene Y. Chen

Abstract: The high cost of data labeling presents a major barrier to deploying machine learning systems at scale. Semi-supervised learning (SSL) mitigates this challenge by utilizing unlabeled data alongside limited labeled examples, while the emergence of foundation models (FMs) offers powerful zero-shot capabilities that can further reduce labeling cost. However, directly fine-tuning large FMs is often impractical in resource-constrained settings, and naïvely using their pseudo-labels for unlabeled data can degrade performance due to its unreliablity or domain mismatch with target task. In this work, we introduce ZeroMatch, a novel SSL framework that integrates knowledge distillation with consistency-based learning to jointly leverage labeled data, unlabeled data, and pseudo-labels from FMs. ZeroMatch trains a compact student model and access FMs only through inference services, making it suitable for low-resource environments such as personal devices with limited compute. Experiments on six vision and language classification benchmarks show that ZeroMatch consistently outperforms standard SSL and zero-shot augmented methods, demonstrating its effectiveness and robustness across a range of foundation model qualities.

Submission Type: Long submission (more than 12 pages of main content)

Changes Since Last Submission:

For addressing additional comments:
- Detailed explanations of auxiliary-head training dynamics are included in Sec. 4.
- Detailed explanations of pseudo-label generation pipeline are included in Sec. 6.1.
- We discussed computational cost trade-offs compared with AdaMatch in Sec. 6.5.4, and discussed possible strategies to reduce the overheads in Appendix C.
- We listed the lack of analysis on API inference cost as a limitation of our work in Appendix E, and discussed related practical issues and future research directions to address them.
- We explicitly mentioned SRD (which Reviewer vqY2 mentioned as a closely related work) and explained the key difference and the novelty of our work in Sec 2.3.

Other changes:
- We included codebase link and acknowledgements.

Code: https://github.com/jichan3751/zeromatch

Assigned Action Editor: ~Hadi_Jamali-Rad1

Submission Number: 6324