Knowledge Distillation: Bad Models Can Be Good Role ModelsDownload PDF

Published: 31 Oct 2022, Last Modified: 28 Jan 2023NeurIPS 2022 AcceptReaders: Everyone
Keywords: Knowledge Distillation, Teacher-Student, Learning Theory, Learning to Sample, Ensembling
TL;DR: We show that a model trained on noisy data can be a good teacher when unlabeled data is ample even when the teacher has noisy predictions.
Abstract: Large neural networks trained in the overparameterized regime are able to fit noise to zero train error. Recent work of Nakkiran and Bansal has empirically observed that such networks behave as “conditional samplers” from the noisy distribution. That is, they replicate the noise in the train data to unseen examples. We give a theoretical framework for studying this conditional sampling behavior in the context of learning theory. We relate the notion of such samplers to knowledge distillation, where a student network imitates the outputs of a teacher on unlabeled data. We show that samplers, while being bad classifiers, can be good teachers. Concretely, we prove that distillation from samplers is guaranteed to produce a student which approximates the Bayes optimal classifier. Finally, we show that some common learning algorithms (e.g., Nearest-Neighbours and Kernel Machines) can often generate samplers when applied in the overparameterized regime.
Supplementary Material: pdf
11 Replies