Born-Again Neural Networks

Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, Anima Anandkumar

Abstract: Knowledge Distillation (KD) consists of transferring “knowledge” from one machine learning model (the teacher) to another (the student). Commonly, the teacher is a high-capacity model with formidab...