ShuffleNorm: A Better Normalization for Semi-supervised Learning

Changrui Chen, Jungong Han, Kurt Debattista

Keywords: Semi-supervised Learning; Normalization

TL;DR: We identified challenges with normalization layers in semi-supervised learning and solved it with straightforward but effective modification.

Abstract: We identify critical challenges with normalisation layers commonly used in fully supervised learning when applied to semi-supervised settings. Specifically, batch normalisation (BN) can experience severe performance degradation when labelled and unlabelled data have mismatched label distributions, due to biased statistical estimation. This results in unstable gradients, hindering the model's ability to converge effectively. While group/layer normalisation (GN/LN) avoids these issues, it lacks the stochastic regularisation provided by BN, leading to weaker generalisation. Poor generalisation, in turn, produces low-quality pseudo-labels, exacerbating confirmation bias. To address these limitations, we propose novel normalisation techniques termed Shuffle Layer normalisation and Shuffle Group normalisation (SLN/SGN) that introduce controllable randomness into LN/GN without increasing model parameters, thus making semi-supervised learning more robust and effective. Through experiments across diverse datasets, including image, text, and audio modalities, we demonstrate that SLN/SGN significantly enhances the performance of state-of-the-art semi-supervised learning algorithms.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1852