Multi-Layer Knowledge Distillation for Continual Semantic Segmentation

Yan Wang, Pengju Xu, Bingye Wang, Haiying Zhao

Abstract: Recently, knowledge distillation has shown promising results for continual semantic segmentation (CSS). Despite their success, several key issues have not been well addressed in existing studies: 1) Difficulty to balance new and old classes, challenged by catastrophic forgetting. 2) Complex information in images is not fully learned and accuracy in CSS needs improvement. To tackle these challenges, we propose a novel Multi-Layer Knowledge Distillation (MLKD) for Continual Semantic Segmentation for Continual Semantic Segmentation. Specifically, we utilize the branch incremental learning module to mitigate catastrophic forgetting by having two branches that freeze the old class parameters and train the new classes. For each branch, we execute the contextual-awareness component to learn the contextual information. Then, we employ a multi-layer knowledge distillation module to comprehensively capture semantic correlations between classes from the intermediate layer and the final output layer. Extensive experiments on two real-world datasets demonstrate the effectiveness of MLKD, where the maximum improvement can reach to 2.7%.