RT2S: A Framework for Learning with Noisy Labels

Indranil Bhattacharya, Ze Ye, Kaushik Pavani, Sunny Dasgupta

Abstract: We introduce Robust Training with Trust Scores (RT2S), a framework to train machine learning classifiers with potentially noisy labels. RT2S calculates a trust score for each training sample, which indicates the quality of its corresponding label. These trust scores are employed as sample weights during training and optionally during threshold optimization. The trust scores are generated from two sources: (i) the model's confidence in the observed label, leveraging out-of-fold prediction scores to detect anomalous labels in the training data, and (ii) the probability of the correct label, ascertained by a Large Language Model with the ability to identify biased label noise. We evaluate RT2S by training machine learning models on 6 product classification datasets that utilize low-quality labels generated by a rule-based classification engine acting as a surrogate labeler. Our experimental findings indicate that RT2S outperforms all baselines, and achieves an average accuracy improvement of 4.38% (max 7.18%) over rule-based classifiers in particular.

External IDs:dblp:conf/cikm/BhattacharyaYPD23