Confidence Difference Reflects Various Supervised Signals in Confidence-Difference Classification

Yuanchao Dai, Ximing Li, Changchun Li

TL;DR: We analyze the noisy problem in confidence difference classification and propose a novel method with consistency regularization.

Abstract: Training a precise binary classifier with limited supervision in weakly supervised learning scenarios holds considerable research significance in practical settings. Leveraging pairwise unlabeled data with confidence differences has been demonstrated to outperform learning from pointwise unlabeled data. We theoretically analyze the various supervisory signals reflected by confidence differences in confidence difference (ConfDiff) classification and identify challenges arising from noisy signals when confidence differences are small. To address this, we partition the dataset into two subsets with distinct supervisory signals and propose a consistency regularization-based risk estimator to encourage similar outputs for similar instances, mitigating the impact of noisy supervision. We further derive and analyze its estimation error bounds theoretically. Extensive experiments on benchmark and UCI datasets demonstrate the effectiveness of our method. Additionally, to effectively capture the influence of real-world noise on the confidence difference, we artificially perturb the confidence difference distribution and demonstrate the robustness of our method under noisy conditions through comprehensive experiments.

Lay Summary:

When training models with limited supervision, comparing pairs of examples often provides better learning signals than labeling individual examples. However, we discover a critical challenge: small confidence differences between paired examples can introduce unreliable signals, as these examples might belong to either the same or different classes, potentially misleading the model during training.

To solve this problem, we develop a method that separates data pairs based on their confidence differences. For pairs with small differences, we apply a special technique called consistency regularization that encourages the model to produce similar outputs, assuming they likely belong to the same class. For pairs with large differences, we maintain their strong guidance signals since they reliably indicate different classes.

Our experiments across multiple datasets demonstrate that this approach consistently outperforms existing methods, even when data contains significant noise. This makes our technique particularly valuable for real-world applications like medical diagnosis, where precise labeling is challenging but comparing patient cases is often more straightforward and reliable.

Primary Area: General Machine Learning->Unsupervised and Semi-supervised Learning

Keywords: weakly supervised classification, error bound, confidence-difference classification

Submission Number: 6776