Mitigating Confirmation Bias in Semi-supervised Learning via Efficient Bayesian Model Averaging
Abstract: State-of-the-art (SOTA) semi-supervised learning (SSL) methods have been highly successful in leveraging a mix of labeled and unlabeled data, often via self-training or pseudo-labeling. During pseudo-labeling, the model's predictions on unlabeled data are used for training and may result in confirmation bias where the model reinforces its own mistakes. In this work, we show that SOTA SSL methods often suffer from confirmation bias and demonstrate that this is often a result of using a poorly calibrated classifier for pseudo labeling. We introduce BaM-SSL, an efficient Bayesian Model averaging technique that improves uncertainty quantification in SSL methods with limited computational or memory overhead. We demonstrate that BaM-SSL mitigates confirmation bias in SOTA SSL methods across standard vision benchmarks of CIFAR-10, CIFAR-100, giving up to 16% improvement in test accuracy on the CIFAR-100 with 400 labels benchmark. Furthermore, we also demonstrate their effectiveness in additional realistic and challenging problems, such as class-imbalanced datasets and in photonics science.
Submission Length: Regular submission (no more than 12 pages of main content)
Changes Since Last Submission: Camera-ready submission. Added minor revisions requested by Action Editor:
- Table 12 in Appendix H showing run-times for all settings
- Extreme low-label settings for CIFAR-100 in a new section of Appendix I, exploring extreme settings with only 1 label and 2 labels per class and showing larger improvements using our BaM approach. Also added clarifications about large variance for different initializations in CIFAR-10 in section 6.1.
Code: https://github.com/clott3/BaM-SSL
Supplementary Material: pdf
Assigned Action Editor: ~Frederic_Sala1
License: Creative Commons Attribution 4.0 International (CC BY 4.0)
Submission Number: 1013
Loading