Non-Stationary Predictions May Be More Informative: Exploring Pseudo-Labels with a Two-Phase Pattern of Training Dynamics

Hongbin Pei, Jingxin Hai, Yu Li, Huiqi Deng, Denghao Ma, Jie Ma, Pinghui Wang, Jing Tao, Xiaohong Guan

TL;DR: In contrast to existing pseudo-labeling methods, this paper explores a new type of pseudo-labels --- predicted labels that do not exhibit high confidence scores and training stationary.

Abstract: Pseudo-labeling is a widely used strategy in semi-supervised learning. Existing methods typically select predicted labels with high confidence scores and high training stationarity, as pseudo-labels to augment training sets. In contrast, this paper explores the pseudo-labeling potential of predicted labels that **do not** exhibit these characteristics. We discover a new type of predicted labels suitable for pseudo-labeling, termed *two-phase labels*, which exhibit a two-phase pattern during training: *they are initially predicted as one category in early training stages and switch to another category in subsequent epochs.* Case studies show the two-phase labels are informative for decision boundaries. To effectively identify the two-phase labels, we design a 2-*phasic* metric that mathematically characterizes their spatial and temporal patterns. Furthermore, we propose a loss function tailored for two-phase pseudo-labeling learning, allowing models not only to learn correct correlations but also to eliminate false ones. Extensive experiments on eight datasets show that **our proposed 2-*phasic* metric acts as a powerful booster** for existing pseudo-labeling methods by additionally incorporating the two-phase labels, achieving an average classification accuracy gain of 1.73% on image datasets and 1.92% on graph datasets.

Lay Summary:

Using model predictions to label unlabeled data—commonly known as pseudo-labeling—is an effective strategy for improving model performance. However, the key challenge lies in selecting reliable pseudo labels to incorporate into the training set.

To address this issue, we analyze training dynamics, which refer to the model’s output over successive training epochs. Our research identifies a previously overlooked type of sample that is well-suited for pseudo-labeling, characterized by two-phase training dynamics: during the early stages of training, the model’s predictions consistently converge on one class, while in later stages, the model’s predictions stabilize in a different class. To capture and utilize this behavior, we propose a 2-*phasic* metric that integrates both temporal and spatial perspectives to efficiently identify such samples.

Through experiments on node classification and image classification tasks, we demonstrate that these samples are not only accurately predicted, but also contain richer class-discriminative information. Our findings reveal a new category of pseudo-label candidates that existing methods tend to ignore, allowing our approach to serve as a booster to current pseudo-labeling algorithms. Furthermore, our work highlights the untapped potential of non-stationary training dynamics as a valuable signal in semi-supervised learning.

Link To Code: https://github.com/XJTU-Graph-Intelligence-Lab/two-phasic-for-pseudo-labeling

Primary Area: Deep Learning->Self-Supervised Learning

Keywords: Pseudo-Labels, Two-phase pattern, Semi-supervised learning, Self-training

Submission Number: 9729