Smooth Pseudo-Labeling

Nikolaos Karaliolios, Hervé Le Borgne, Florian Chabot

Abstract:

Semi-Supervised Learning (SSL) seeks to leverage large amounts of non-annotated data along with the smallest amount possible of annotated data in order to achieve the same level of performance as if all data were annotated. A fruitful method in SSL is Pseudo-Labeling (PL), which, however, suffers from the important drawback that the associated loss function has discontinuities in its derivatives, which cause instabilities in performance when labels are very scarce.
In the present work, we address this drawback with the introduction of a Smooth Pseudo-Labeling ($SPL$) loss function. It consists in adding a multiplicative factor in the loss function that smooths out the discontinuities in the derivative due to thresholding. In our experiments, we test our improvements on FixMatch and show that it significantly improves the performance in the regime of scarce labels, without addition of any modules, hyperparameters, or computational overhead. In the more stable regime of abundant labels, performance remains at the same level. Robustness with respect to variation of hyperparameters and training parameters is also significantly improved. Moreover, we introduce a new benchmark, where labeled images are selected randomly from the whole dataset, without imposing representation of each class proportionally to its frequency in the dataset. We see that the smooth version of FixMatch does appear to perform better than the original, non-smooth implementation. However, more importantly, we notice that both implementations do not necessarily see their performance improve when labeled images are added, an important issue in the design of SSL algorithms that should be addressed so that Active Learning algorithms become more reliable and explainable.

Submission Length: Long submission (more than 12 pages of main content)

Changes Since Last Submission: Small addition to the abstract. The introduction and the related work have been reworked with additions as requested by the reviewers. Notation and arguments in the method section have been changed in order to improve clarity. Some experiments were moved from the appendix to the main text.

Assigned Action Editor: ~Tongliang_Liu1

Submission Number: 1945