PURSUhInT: In Search of Informative Hint Points Based on Layer Clustering for Knowledge Distillation

Reyhan Kevser Keser, Aydin Ayanzadeh, Omid Abdollahi Aghdam, Çaglar Kilcioglu, Behcet Ugur Töreyin, Nazim Kemal Ure

Abstract: Highlights•We show that selection of hint points impacts the performance of distillation.•We propose a clustering-based hint selection method.•Our method can be applied to any type of distillation that uses at least two hints.•Our method is applicable for any student, once it is applied on a chosen teacher.