Extracting Latent Generalization from Models Trained with Noisy Labels

Simran Ketha, Venkatakrishnan Ramaswamy

Keywords: Memorization, Generalization, Deep Neural Networks

Abstract: Overparameterized Deep Neural Networks that generalize well underlie much of the recent successes in Deep Learning. It has also been known that when training data labels are noisy, Deep Networks, on training, exhibit the capacity to learn this label noise, which hurts their generalization, as manifested by degraded test accuracies. Here, we investigate whether we can extract more reliable predictions from these models whose predictive power is impacted by such unreliable training data, while sticking to the standard training paradigm. Specifically, we consider the question of extracting better generalization from the latent representations of the layers of the model, in this setting. To this end, we study the class-conditional subspaces corresponding to the training data corrupted with label noise. Furthermore, we examined the geometry of the layerwise outputs in relation to these subspaces. We find, surprisingly, that doing so leads to a technique to extract significantly better generalization than provided by the corresponding model. We show results exemplifying this phenomenon on multiple models trained with a number of standard datasets. Our work demonstrates that we can extract underutilized latent generalization present in the internal representations of models trained with data which has label noise.

Submission Number: 159