Towards Understanding Knowledge Distillation

Mary Phuong, Christoph Lampert

Abstract: Knowledge distillation, i.e., one classifier being trained on the outputs of another classifier, is an empirically very successful technique for knowledge transfer between classifiers. It has even ...