A Curriculum View of Robust Loss Functions

Zebin Ou, Jiao Xue, Qiankun Fu, Yue Zhang

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: robust loss function, deep learning, noisy label

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: Robust loss functions are designed to mitigate the adverse impacts of label noise, which enjoy theoretical guarantees that is agnostic to the training dynamics. However, these guarantees fail to characterize some empirical phenomenons. To address this limitation, we unify a broad array of loss functions into a novel standard form, which consists of a primary loss function inducing a sample-weighting curriculum and an optional implicit regularizer. The resulting curriculum view leads to a straightforward analysis of the training dynamics, which helps demystify how loss functions and regularizers affect learning and noise robustness. In particular, we show that robust loss functions implicitly sift and neglect corrupted samples. We then analyze the roles of regularizers with different loss functions. Finally, we dissect the cause of the notorious underfitting issue and provide effective fixes.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3009