Stingy Teacher: Sparse Logits Suffice to Fail Knowledge DistillationDownload PDF

29 Sept 2021, 00:33 (edited 05 Oct 2021)ICLR 2022 Conference Withdrawn SubmissionReaders: Everyone
  • Keywords: Knowledge Distillation, avoid knowledge leaking
  • Abstract: Knowledge distillation (KD) aims to transfer the discrimination power of pre-trained teacher models to (more lightweight) student models. However, KD also poses the risk of intellectual properties (IPs) leakage of teacher models. Even if the teacher model is released as a black box, it can still be cloned through KD by imitating input-output behaviors. To address this unwanted effect of KD, the concept of Nasty Teacher was proposed recently. It is a special network that achieves nearly the same accuracy as a normal one, but significantly degrades the accuracy of student models trying to imitate it. Previous work builds the nasty teacher by retraining a new model and distorting its output distribution from the normal one via an adversarial loss. With this design, the ``nasty" teacher tends to produce sparse and noisy logits. However, it is unclear why the distorted distribution of the logits is catastrophic to the student model. In addition, the retraining process used in Nasty Teacher is undesirable, not only degrading the performance of the teacher model but also limiting its applicability to large datasets. In this paper, we provide a theoretical analysis of why the sparsity of logits is key to Nasty Teacher. We further propose Stingy Teacher, a much simpler yet more effective algorithm to prevent imitation through KD without incurring accuracy drop or requiring retraining. Stingy Teacher directly manipulates the logits of a standard pre-trained network by maintaining the values for a small subset of classes while zeroing out the rest. Extensive experiments on large-scale datasets and various teacher-student pairs demonstrate that our stingy teacher is highly effective and more catastrophic to student models than the Nasty Teacher. Code and pre-trained models will be released upon acceptance.
  • One-sentence Summary: We theoretically analyze sparse logits can make knowledge distillation fail, and propose Stingy Teacher with just a simple manipulation of the output logits to prevent unauthorized cloning through knowledge distillation
  • Supplementary Material: zip
5 Replies