On Overcompression in Continual Semantic Segmentation

Maciej Kowalski, Thomas L Lee, Amos Storkey

Keywords: Continual Learning, Class-Incremental Semantic Segmentation, Information Bottleneck, overcompression, dropout

Abstract: Class-Incremental Semantic Segmentation (CISS) is an emerging challenge of Continual Learning (CL) in Computer Vision. In addition to the well-known issue of catastrophic forgetting, CISS suffers from the semantic drift of the background class, further increasing forgetting. Existing attempts aim to solve this using pseudo-labelling, knowledge distillation or model freezing. We argue and demonstrate that frozen or rigid models suffer from poor expressibility due to overcompression. We improve on these methods by focusing on the offline training process and the expressiveness of the learnt representations. Beyond the characterisation and demonstration of this issue in terms of the Information Bottleneck principle, we show the benefit of two practical measures: (1) using shared but wider convolution modules before final classifiers to improve scaling for new, continual tasks; (2) introducing dropout into the encoder-decoder architecture to improve regularisation and decrease the overcompression of information in the representation space. We improve the IoU on the 15-1 and 10-1 scenarios by over 2% and 3% respectively while maintaining a smaller memory and MAdds footprint. Last, we propose a new benchmark setting that lies closer to the nature of lifelong learning to drive the development of more realistic and valuable architectures in the future.

TL;DR: We improve the expressiveness of encoder modules to show that Continual Semantic Segmentation models suffer from overcompression.