Factorized and progressive knowledge distillation for CTC-based ASR models

Sanli Tian, Zehan Li, Zhaobiao Lyv, Gaofeng Cheng, Qing Xiao, Ta Li, Qingwei Zhao

Abstract: Highlights•We explore why the conventional KD underperforms when applied to CTC models.•We propose Factorized KL-divergence for CTC-based models’ KD.•We propose a progressive KD framework to gradually build up the student’s knowledge.