Training Over-parameterized Models with Non-decomposable ObjectivesDownload PDF

21 May 2021, 20:45 (modified: 22 Jan 2022, 16:07)NeurIPS 2021 PosterReaders: Everyone
Keywords: Non-decomposable objectives, Class imbalance, Cost-sensitive Losses, Over-parameterized Models, Logit-adjustment, Margin, Constraints
TL;DR: New cost-sensitive losses for training over-parameterized models with complex objectives
Abstract: Many modern machine learning applications come with complex and nuanced design goals such as minimizing the worst-case error, satisfying a given precision or recall target, or enforcing group-fairness constraints. Popular techniques for optimizing such non-decomposable objectives reduce the problem into a sequence of cost-sensitive learning tasks, each of which is then solved by re-weighting the training loss with example-specific costs. We point out that the standard approach of re-weighting the loss to incorporate label costs can produce unsatisfactory results when used to train over-parameterized models. As a remedy, we propose new cost- sensitive losses that extend the classical idea of logit adjustment to handle more general cost matrices. Our losses are calibrated, and can be further improved with distilled labels from a teacher model. Through experiments on benchmark image datasets, we showcase the effectiveness of our approach in training ResNet models with common robust and constrained optimization objectives.
Supplementary Material: pdf
Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.
Code: https://github.com/google-research/google-research/tree/master/non_decomp
18 Replies

Loading