Derivative Manipulation for General Example Weighting

Xinshao Wang, Elyor Kodirov, Yang Hua, Neil M. Robertson

Keywords: loss function, gradient descent, roubst deep learning, example weighting, regularization, label noise, sample imbalance

Abstract:

Real-world large-scale datasets usually contain noisy labels and are imbalanced. Therefore, we propose derivative manipulation (DM), a novel and general example weighting approach for training robust deep models under these adverse conditions.

DM has two main merits. First, loss function and example weighting are common techniques in the literature. DM reveals their connection (a loss function does example weighting) and is a replacement of both. Second, despite that a loss defines an example weighting scheme by its derivative, in the loss design, we need to consider whether it is differentiable. Instead, DM is more flexible by directly modifying the derivative so that a loss can be a non-elementary format too. Technically, DM defines an emphasis density function by a derivative magnitude function. DM is generic in that diverse weighting schemes can be derived.

Extensive experiments on both vision and language tasks prove DM’s effectiveness.

One-sentence Summary: Loss function and example weighting are common techniques. DM reveals their connection (a loss function does example weighting) and is a replacement of both.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=B2oMPrqzmg