Recruiting the Best Teacher Modality: A Customized Knowledge Distillation Method for if Based Nephropathy Diagnosis

Ning Dai, Lai Jiang, Yibing Fu, Sai Pan, Mai Xu, Xin Deng, Pu Chen, Xiangmei Chen

Abstract: The joint use of multiple imaging modalities for medical image has been widely studied in recent years. The fusion of information from different modalities has demonstrated the performance improvement for some medical tasks. For nephropathy diagnosis, immunofluorescence (IF) is one of the most widely-used medical image due to its ease of acquisition with low cost, which is also an advanced multi-modality technique. However, the existing methods mainly integrate information from diverse sources by averaging or combining them, failing to exploit multi-modality knowledge in details. In this paper, we observe that the 7 modalities of IF images have different impact on different nephropathy categories. Accordingly, we propose a knowledge distillation framework to transfer knowledge from the trained single-modality teacher networks to a multi-modality student network. On top of this, given a input IF sequence, a recruitment module is developed to dynamically assign weights to teacher models and optimize the performance of student model. By applying on several different architectures, the extensive experimental results verify the effectiveness of our method for nephropathy diagnosis.