DDDG: A dual bi-directional knowledge distillation method with generative self-supervised pre-training and its hardware implementation on SoC for ECG

Huaicheng Zhang, Wenhan Liu, Qianxi Guo, Jiguang Shi, Sheng Chang, Hao Wang, Jin He, Qijun Huang

Abstract: Highlights•Knowledge distillation and generative self-supervised learning are incorporated.•Dual distillations are contained in the stage of pre-training and fine-tuning.•Bi-directional knowledge distillation enhance teacher models in reverse.•Ultra-lightweight and well-performing student models are obtained.•Heterogeneously deploy models on resource-limited devices for real-time inference.