LLS: Regulating Neural Network Training via Learnable Label Smoothing

Sachin Chhabra, PrasanthSai Gouripeddi, Hemanth Venkateswara, Baoxin Li

Keywords: Learnable Label Smoothing, LLS, Label Regularization

TL;DR: Improved label smoothing that does not impact inter-class relationship and learns the optimal assignment.

Abstract:

Training a neural network using one-hot targets often leads to the issue of overconfidence.
To address this, Label Smoothing has been introduced, modifying the targets to a mix of one-hot encoding and a uniform probability vector.
However, the uniform probability vector indiscriminately assigns equal weights to all categories, thereby undermining inter-category relationships. To overcome these challenges, we propose a novel solution, Learnable Label Smoothing (LLS) that aims to regulate training by granting networks the ability to assign optimal targets. Unlike conventional methods, Learnable Label Smoothing utilizes probability vectors unique to each category, resulting in diverse targets. The acquired relationships are beneficial for regularization and also prove to be transferable, facilitating knowledge distillation even in the absence of a Teacher model. Our extensive experiments across multiple datasets highlight the advantages of our method in addressing both overconfidence and the preservation of inter-category relationships in neural network training.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 11381