Focus on the Fog: Leveraging Student Uncertainty for Guided Knowledge Distillation in Semantic Segmentation

Emil Mededovic, Fabian Gülhan, Rüveyda Yilmaz, Johannes Stegmaier

Keywords: Semantic Segmentation, Knowledge Distillation, Uncertainty Estimation

Abstract: Current knowledge distillation (KD) methods for semantic segmentation focus on distilling the teacher's knowledge via logit and feature-based techniques. Recent work explored the improvement of knowledge distillation methods by incorporating the uncertainty of the teacher in dense prediction tasks, primarily in object detection. Yet, its application in knowledge distillation for semantic segmentation has received limited attention. Moreover, utilizing the uncertainty on the student side remains largely underexplored. We posit that student-side uncertainty can serve as a valuable signal for guiding the distillation process in semantic segmentation. To this end, we propose Focus on the Fog (FOTF), a novel uncertainty-guided distillation approach that estimates and leverages student-side uncertainty during training. Specifically, we formulate an uncertainty‑weighted distillation loss for semantic segmentation that is dynamically modulated by the student’s uncertainty, estimated via Monte Carlo Dropout. This amplifies the distillation signal in spatial regions and semantic classes where the student model exhibits low certainty, thereby providing more targeted guidance during training. Extensive experiments on the Cityscapes, CamVid and Pascal VOC datasets demonstrate the effectiveness of our method, both as a standalone technique and as an add-on to existing state-of-the-art knowledge distillation methods. The code will be made publicly available upon acceptance.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 17843