EWC-Guided Diffusion Replay for Exemplar-Free Continual Learning in Medical Imaging

Anoushka Harit, William Prew, Zhongtian Sun, Florian Markowetz

Keywords: Continual Learning, Generative Replay, Diffusion Models, Elastic Weight Consolidation

TL;DR: We combine diffusion-based generative replay withElastic Weight Consolidation to enable exemplar-free, privacy-preserving continual learning for medical imaging foundation models.

Abstract: Medical imaging foundation models must adapt continually, but retraining is limited by privacy and cost. We propose an exemplar-free framework combining class-conditional diffusion replay (DDPMs) with synaptic stability from Elastic Weight Consolidation (EWC). A compact Vision Transformer backbone is evaluated across eight MedMNIST v2 tasks and CheXpert. Our method attains 0.851 AUROC on CheXpert, cuts forgetting by over 30\% versus DER++, and approaches joint training (0.869), while preserving privacy and efficiency. Analysis links forgetting to replay fidelity and parameter stability, underscoring the complementary roles of DDPM and EWC. This establishes a scalable, privacy-preserving route for continual FM adaptation.

Serve As Reviewer: ~Anoushka_Harit1

Submission Number: 12