Improving realistic semi-supervised learning with doubly robust estimation

ICLR 2026 Conference Submission25185 Authors

Keywords: semi-supervised learning, doubly robust estimation

TL;DR: We use doubly robust estimation to improve the class distribution estimation and classification accuracy for distribution mismatch settings in semi-supervised learning

Abstract:

A major challenge in Semi-Supervised Learning (SSL) is the mismatch between the labeled and unlabeled class distributions.
Most successful SSL approaches are based on pseudo-labeling of the unlabeled data, and therefore are susceptible to confirmation bias because the classifier being trained is biased towards the labeled class distribution and thus performs poorly on unlabeled data.
While distribution alignment alleviates this bias, we find that the distribution estimation at the end of training can still be improved with the doubly robust estimator, a theoretically sound approach that derives from semi-parametric efficiency theory.
As a result, we propose a 2-stage approach where we first train an SSL classifier but only use this initial prediction for the doubly robust estimator of the class distribution, and then train a second SSL classifier but fixing the improved distribution estimation from the start.
For training the classifier, we use a principled expectation-maximization framework for SSL with label shift, showing that the popular distribution alignment heuristic improves the data log-likelihood in the E-step, and that this EM is equivalent to the recent SimPro algorithm after reparameterization and logit adjustment but is much older and more interpretable (using the missingness mechanism).
Experimental results demonstrate the improved class distribution estimation of the doubly robust estimator and subsequent improved classification accuracy with our 2-stage approach.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 25185