Enhancing Diabetic Retinopathy Detection Through Transformer Based Knowledge Distillation and Explainable AI

Abdullah Al Shafi, Miraj Hossain Shawon, Nida Shahid, Rita Rahman, Riasat Khan

Abstract: Diabetic retinopathy (DR) is a critical complication of diabetes and is characterized by damage to retinal blood vessels. Without early diagnosis and treatment, it can lead to vision loss. The objective of this study is to investigate diverse methodologies to address the class imbalance issue in the retinal fundus image dataset and apply transformer-based knowledge distillation (KD) for efficient DR detection. Various deep-learning models have been implemented to classify RGB fundus images into five distinct classes of DR. The retinal fundus images of the employed APTOS dataset were classified using the following models: ResNet34, DenseNet121, GoogLeNet MobileViTv2, DeiT3, and KD. Initial training on the imbalanced APTOS dataset showed promising results, but to explore a more robust approach, two data balancing approaches were used, namely cost-sensitive learning and data augmentation. It was found that MobileViTv2 achieved the best results with the APTOS cost-sensitive learning approach, achieving a kappa score of 0.94 and a macro F1 score of 0.73. Furthermore, the KD technique with MobileViTv2 as a teacher and GoogLeNet as a student model achieved a WKS of 0.90 with significantly fewer parameters than the teacher model. Finally, the automatic predictions of the deep learning models are interpreted using the SHAP explainable AI framework. These findings suggest that the cost-sensitive approach and lightweight KD model can substantially enhance the performance of the implemented deep learning models for DR detection in memory-constrained healthcare devices.