Respecting Transfer Gap in Knowledge DistillationDownload PDF

Published: 31 Oct 2022, Last Modified: 22 Oct 2023NeurIPS 2022 AcceptReaders: Everyone
Keywords: knowledge distillation, transfer gap, inverse probability weighting
TL;DR: We propose Inverse Probability Weighting Distillation (IPWD) to tackle the imbalanced knowledge of the teacher model on the transfer set during knowledge distillation.
Abstract: Knowledge distillation (KD) is essentially a process of transferring a teacher model's behavior, e.g., network response, to a student model. The network response serves as additional supervision to formulate the machine domain, which uses the data collected from the human domain as a transfer set. Traditional KD methods hold an underlying assumption that the data collected in both human domain and machine domain are both independent and identically distributed (IID). We point out that this naive assumption is unrealistic and there is indeed a transfer gap between the two domains. Although the gap offers the student model external knowledge from the machine domain, the imbalanced teacher knowledge would make us incorrectly estimate how much to transfer from teacher to student per sample on the non-IID transfer set. To tackle this challenge, we propose Inverse Probability Weighting Distillation (IPWD) that estimates the propensity of a training sample belonging to the machine domain, and assigns its inverse amount to compensate for under-represented samples. Experiments on CIFAR-100 and ImageNet demonstrate the effectiveness of \ours~for both two-stage distillation and one-stage self-distillation.
Supplementary Material: pdf
Community Implementations: [![CatalyzeX](/images/catalyzex_icon.svg) 2 code implementations](https://www.catalyzex.com/paper/arxiv:2210.12787/code)
20 Replies

Loading