Regularizing Predictions via Class-wise Self-knowledge Distillation

Sukmin Yun, Jongjin Park, Kimin Lee, Jinwoo Shin

TL;DR: We propose a new regularization technique based on the knowledge distillation.

Abstract: Deep neural networks with millions of parameters may suffer from poor generalizations due to overfitting. To mitigate the issue, we propose a new regularization method that penalizes the predictive distribution between similar samples. In particular, we distill the predictive distribution between different samples of the same label and augmented samples of the same source during training. In other words, we regularize the dark knowledge (i.e., the knowledge on wrong predictions) of a single network, i.e., a self-knowledge distillation technique, to force it output more meaningful predictions. We demonstrate the effectiveness of the proposed method via experiments on various image classification tasks: it improves not only the generalization ability, but also the calibration accuracy of modern neural networks.

Keywords: regularization, knowledge distillation