Maximizing discrimination capability of knowledge distillation with energy function

Seonghak Kim, Gyeongdo Ham, Suin Lee, Donggon Jang, Daeshik Kim

Abstract: Highlights•We confirm that sample’s energy is crucial for the model’s discrimination ability.•Energy-dependent temperature is utilized to transfer the teacher’s optimal knowledge.•Augmenting high-energy images yields high performance with lower computational costs.•Energy-based methods boost performance on challenging dataset, showing practicality.