Training Semi-Supervised Deep Learning Models with Heuristic Early Stopping Rules

Jialin Wu, Soohyun Kim, Younghoon Kim, Samprit Banerjee

Keywords: semi-supervised deep learning, neural network, convergence, generalizability, predictive modeling, model optimization

Abstract: Semi-supervised learning (SSL), especially when combined with deep learning (DL) models, is a useful technique when there is a substantial amount of unlabeled data. This is particularly relevant in healthcare applications, such as mHealth, where data is often collected through smartphones. Labels are typically obtained via self-reported questions delivered by the device and tend to have a high rate of non-response i.e., missing labels. Despite its benefit, there is a lack of objective methodology on how to train semi-supervised deep learning (SSDL) models. In this study, we propose a framework for early-stopping in SSDL that terminates learning to prevent overfitting and before the performance starts to deteriorate. Our approach focuses on three aspects: model stability, generalizability, and high-confidence pseudo-label (i.e., label assigned to unlabeled data during SSL). We first monitor changes in learned weights of the model to assess convergence, using weight stabilization. We also track cross-entropy loss, identifying which iteration of the SSL algorithm minimizes validation loss and improves generalizability. Lastly, we use a sliding window method to assess our confidence in the pseudo-labels, retaining only the most reliable labels during training. Combining these criteria, this SSDL framework can be used to train deep learning models in the context of SSL with an objective criteria that prevents overfitting and improves generalizability. We apply this SSDL training strategy to mHealth data (device sensor data and self-reported data) collected from participants in a clinical trial, which consists of 4,700 observations, 62% of which are unlabeled. Using this objective early stopping criteria for training, we achieve improvements in accuracy and F1 scores, compared to the benchmark model where the early stopping criteria is not applied.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 11680