Symmetrization of Loss Functions for Robust Training of Neural Networks in the Presence of Noisy Labels
Keywords: Noisy labels, Symmetric loss functions, Multi-class loss decomposition, Unhinged loss function
TL;DR: We investigate a novel method to create symmetric multi-class loss functions, introducing SGCE, smoothly transitioning between the multi-class unhinged loss and the mean absolute error.
Abstract: Labeling a training set is not only often expensive but also susceptible to errors. Consequently, the development of robust loss functions to handle label noise has emerged as a problem of great importance. The symmetry condition provides theoretical guarantees for robustness to such noise. In this work, we investigate a symmetrization method that arises from the unique decomposition of any multi-class loss function into a sum of a symmetric loss function and a class-insensitive term. Notably, the special case of symmetrizing the cross-entropy loss leads to a multi-class extension of the unhinged loss function. This loss function is linear, but unlike in the binary case, it must have specific coefficients in order to satisfy the symmetry condition. Under appropriate assumptions, we demonstrate that this multi-class unhinged loss function is the unique convex multi-class symmetric loss function. It holds a significant role among multi-class symmetric loss functions since the linear approximation of any symmetric loss function around points with equal components must be equivalent to the multi-class unhinged. Furthermore, we introduce SGCE and α-MAE, two novel loss functions that smoothly transition between the multi-class unhinged loss and the Mean Absolute Error (MAE). Our experiments demonstrate superior performance over previous state-of-the-art robust loss functions on standard benchmarks, highlighting the effectiveness of our approach in handling label noise.
Primary Area: learning theory
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 12228
Loading