FOCUS: Feature Replay with Optimized Channel-Consistent Dropout for U-Net Skip-Connections

Simon Johannes Joham, Franz Thaler, Arnela Hadzic, Martin Urschler

Abstract: We address image segmentation in the domain-incremental continual learning scenario, a use-case frequently encountered in medical diagnostics where privacy regulations and storage constraints prevent access to historical data. In this scenario, segmentation models must learn to cope with new domains (e.g., difference in imaging protocols or patient population) while maintaining performance on previously learned domains without full access to past data. Feature-based replay addresses the privacy concerns by only storing latent feature representations instead of original images. However, existing feature replay approaches have a critical limitation: they sacrifice U-Net skip-connections, which are essential for achieving high segmentation accuracy and fast convergence. This limitation significantly impacts clinical viability, especially when alternatives such as full model retraining or maintaining domain-specific models are available. Therefore, we propose feature replay with optimized channel-consistent dropout for U-Net skip-connections (FOCUS). FOCUS enables crucial skip-connections in feature replay while respecting privacy and storage constraints, and integrates recent domain generalization techniques based on data augmentation. Evaluation across two domain-incremental continual MRI segmentation settings demonstrates that FOCUS achieves substantial improvements (up to 21% average DSC) over existing methods, while saving only 0.5% of the original feature information per domain. The code is available at https://github.com/imigraz/FOCUS/.

External IDs:dblp:conf/miccai/JohamTHU25