Zero-Shot Knowledge Distillation in Deep Networks

Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj, Venkatesh Babu Radhakrishnan, Anirban Chakraborty

Abstract: Knowledge distillation deals with the problem of training a smaller model (<em>Student</em>) from a high capacity source model (<em>Teacher</em>) so as to retain most of its performance. Existing a...