Searching to Exploit Memorization Effect in Learning from Corrupted Labels

Hansi Yang, Quanming Yao, Bo Han, Gang Niu

TL;DR: Using automated machine learning techniques to exploit memorization effect in learning from corrupted labels

Abstract: Sample-selection approaches, which attempt to pick up clean instances from the training data set, have become one promising direction to robust learning from corrupted labels. These methods all build on the memorization effect, which means deep networks learn easy patterns first and then gradually over-fit the training data set. In this paper, we show how to properly select instances so that the training process can benefit the most from the memorization effect is a hard problem. Specifically, memorization can heavily depend on many factors, e.g., data set and network architecture. Nonetheless, there still exists general patterns of how memorization can occur. These facts motivate us to exploit memorization by automated machine learning (AutoML) techniques. First, we designed an expressive but compact search space based on observed general patterns. Then, we propose to use the natural gradient-based search algorithm to efficiently search through space. Finally, extensive experiments on both synthetic data sets and benchmark data sets demonstrate that the proposed method can not only be much efficient than existing AutoML algorithms but can also achieve much better performance than the state-of-the-art approaches for learning from corrupted labels.

Keywords: Noisy Label, Deep Learning, Automated Machine Learning

Community Implementations: [ 3 code implementations](https://www.catalyzex.com/paper/arxiv:1911.02377/code)