Hard Samples, Bad Labels: Robust Loss Functions That Know When to Back Off

TMLR Paper7744 Authors

Abstract: Incorrectly labelled training data are frustratingly ubiquitous in both benchmark and specially curated datasets. Such mislabelling clearly adversely affects the performance and generalizability of models trained through supervised learning on the associated datasets. Frameworks for detecting label errors typically require well-trained / well-generalized models; however, at the same time most frameworks rely on training these models on corrupt data, which clearly has the effect of reducing model generalizability and subsequent effectiveness in error detection – unless a training scheme robust to label errors is employed. We propose two novel loss functions, Blurry Loss and Piecewise-zero Loss, that enhance robustness to label errors by de-weighting or disregarding difficult-to-classify samples, which are likely to be erroneous. These loss functions leverage the idea that mislabelled examples typically appear as outliers to their as-labelled class, being difficult to classify, and should contribute less to the learning signal. Comprehensive experiments on a variety of both artificially corrupted and real-world datasets demonstrate that the proposed loss functions outperform state-of-the-art robust loss functions in nearly all cases, achieving superior F1 and Balanced Accuracy scores for error detection. Further analyses through ablation studies offer insights to confirm the mechanism through which these loss functions operate, and demonstrate their broad applicability to cases of both uniform and non-uniform corruption, and with different label error detection frameworks. By using these robust loss functions, machine learning practitioners can more effectively identify, prune, or correct errors in their training data. Code, including a working demonstration Jupyter Notebook, is available at https://anonymous.4open.science/r/Robust\_Loss-6BAD/.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Yuheng_Jia1

Submission Number: 7744