Label Noise: Ignorance Is Bliss

Yilun Zhu, Jianxin Zhang, Aditya Gangrade, Clayton Scott

Keywords: label noise; multi-class classification; learning theory; domain adaptation; minimax rate; self-supervised learning

Abstract:

We establish a new theoretical framework for learning under multi-class, instance-dependent label noise.
This framework casts learning with label
noise as a form of domain adaptation, in particular, domain adaptation
under posterior drift.
We introduce the concept of \emph{relative signal strength} (RSS), a pointwise measure that quantifies the transferability from noisy to clean posterior.
Using RSS, we establish nearly matching upper and lower bounds on the excess risk.
Our theoretical findings support
the simple
\emph{Noise Ignorant Empirical Risk Minimization (NI-ERM)} principle,
which minimizes empirical risk while ignoring label noise.
Finally, we translate this theoretical insight into practice: by
using NI-ERM to fit a linear classifier on top of a self-supervised
feature extractor, we achieve state-of-the-art performance on the
CIFAR-N data challenge.

Primary Area: Learning theory

Submission Number: 12823