Open Peer Review. Open Publishing. Open Access. Open Discussion. Open Directory. Open Recommendations. Open API. Open Source.
Transferring Knowledge to Smaller Network with Class-Distance Loss
Seung Wook Kim, Hyo-Eun Kim
Feb 16, 2017 (modified: Mar 14, 2017)ICLR 2017 workshop submissionreaders: everyone
Abstract:Training a network with small capacity that can perform as well as a larger capacity network is an important problem that needs to be solved in real life applications which require fast inference time and small memory requirement. Previous approaches that transfer knowledge from a bigger network to a smaller network show little benefit when applied to state-of-the-art convolutional neural network architectures such as Residual Network trained with batch normalization. We propose class-distance loss that helps teacher networks to form densely clustered vector space to make it easy for the student network to learn from it. We show that a small network with half the size of the original network trained with the proposed strategy can perform close to the original network on CIFAR-10 dataset.
Enter your feedback below and we'll get back to you as soon as possible.