Data Parameters: A New Family of Parameters for Learning a Differentiable Curriculum

Shreyas Saxena, Oncel Tuzel, Dennis DeCoste

Abstract: Recent works have shown that learning from easier instances first can help Deep Neural Networks (DNNs) generalize better. Knowing which data to present during different stages of training is a challenging problem. In this work, we propose an optimization framework to learn a curriculum over classes or individual training samples in a dataset. We equip each sample and class with a learnable parameter (temperature parameters), which governs its importance in the learning process. During training, at each iteration, as we update the model parameters, we also update the temperature parameters. These updates are done by gradient descent and do not require hand-crafted rules or design. We present extensive evaluation of the proposed method and compare with the state-of-the-art. When applied to image classification task on CIFAR10, CIFAR100 and ImageNet datasets, and object detection task on KITTI dataset, learning a dynamic curriculum leads to consistent gains. When applied to a noisy dataset, the proposed method learns to learn from clean images and improves the state-of-the-art methods by 14%.

CMT Num: 5946

Code Link: https://github.com/apple/ml-data-parameters