Better Generalization with On-the-fly Dataset Denoising

Jiaming Song, Tengyu Ma, Michael Auli, Yann Dauphin

Abstract: Memorization in over-parameterized neural networks can severely hurt generalization in the presence of mislabeled examples. However, mislabeled examples are to hard avoid in extremely large datasets. We address this problem using the implicit regularization effect of stochastic gradient descent with large learning rates, which we find to be able to separate clean and mislabeled examples with remarkable success using loss statistics. We leverage this to identify and on-the-fly discard mislabeled examples using a threshold on their losses. This leads to On-the-fly Data Denoising (ODD), a simple yet effective algorithm that is robust to mislabeled examples, while introducing almost zero computational overhead. Empirical results demonstrate the effectiveness of ODD on several datasets containing artificial and real-world mislabeled examples.

Keywords: dataset denoising, supervised learning, implicit regularization

TL;DR: We introduce a fast and easy-to-implement algorithm that is robust to dataset noise.

Data: [WebVision](https://paperswithcode.com/dataset/webvision-database)