Open Peer Review. Open Publishing. Open Access. Open Discussion. Open Directory. Open Recommendations. Open API. Open Source.
Sample Importance in Training Deep Neural Networks
Tianxiang Gao, Vladimir Jojic
Nov 04, 2016 (modified: Jan 23, 2017)ICLR 2017 conference submissionreaders: everyone
Abstract:The contribution of each sample during model training varies across training iterations and the model's parameters. We define the concept of sample importance as the change in parameters induced by a sample. In this paper, we explored the sample importance in training deep neural networks using stochastic gradient descent. We found that "easy" samples -- samples that are correctly and confidently classified at the end of the training -- shape parameters closer to the output, while the "hard" samples impact parameters closer to the input to the network. Further, "easy" samples are relevant in the early training stages, and "hard" in the late training stage. Further, we show that constructing batches which contain samples of comparable difficulties tends to be a poor strategy compared to maintaining a mix of both hard and easy samples in all of the batches. Interestingly, this contradicts some of the results on curriculum learning which suggest that ordering training examples in terms of difficulty can lead to better performance.
Keywords:Deep learning, Supervised Learning
Enter your feedback below and we'll get back to you as soon as possible.