Holistic Continual Learning under Concept Drift with Adaptive Memory Realignment

Alif Ashrafee, Jędrzej Kozal, Michał Woźniak, Bartosz Krawczyk

Abstract: Traditional continual learning methods prioritize knowledge retention and focus primarily on mitigating catastrophic forgetting, implicitly assuming that the data distribution of previously learned tasks remains static. This overlooks the dynamic nature of real-world data streams, where concept drift permanently alters previously seen data and demands both stability and rapid adaptation. We introduce a holistic framework for continual learning under concept drift that simulates realistic scenarios by evolving task distributions. As a baseline, we consider Full Relearning (FR), in which the model is retrained from scratch on newly labeled samples from the drifted distribution. While effective, this approach incurs substantial annotation and computational overhead. To address these limitations, we propose Adaptive Memory Realignment (AMR), a lightweight alternative that equips rehearsal-based learners with a drift-aware adaptation mechanism. AMR selectively removes outdated samples of drifted classes from the replay buffer and repopulates it with a small number of up-to-date instances, effectively realigning memory with the new distribution. This targeted resampling matches the performance of FR while reducing the need for labeled data and computation by orders of magnitude. To enable reproducible evaluation, we introduce four concept drift variants of standard vision benchmarks: Fashion-MNIST-CD, CIFAR10-CD, CIFAR100-CD, and Tiny-ImageNet-CD, where previously seen classes reappear with shifted representations. Comprehensive experiments on these datasets using several rehearsal-based baselines show that AMR consistently counters concept drift, maintaining high accuracy with minimal overhead. These results position AMR as a scalable solution that reconciles stability and plasticity in non-stationary continual learning environments. Full implementation of our framework and concept drift benchmark datasets are available at: https://github.com/AlifAshrafee/CL-Under-Concept-Drift.

Submission Type: Long submission (more than 12 pages of main content)

Changes Since Last Submission:

- Diagrams resized: Some of the diagrams have been expanded to improve visuals and readability.
- Expanded Limitations section: Added explicit failure case scenarios where AMR may exhibit degraded performance, including adversarial drift concealment, gradual or mixture drift, and open-set recurrence.
- Real-world drift experiments (Appendix B): Added evaluation on the CLEAR-10 dataset with natural temporal drift, including discussion of drift detection challenges (uncertainty-based vs. MMD) on gradual temporal shifts.
- Backbone robustness experiments (Appendix C): Added experiments with ResNet-152 and ViT-S backbones on S-CIFAR10-CD and S-CIFAR100-CD to validate that AMR's effectiveness is architecture-independent.
- Main text references: Added references in Section 4 (Experimental Setup) to the new appendix sections for real-world experiments and backbone.
- Appendix formatting: Consolidated hyperparameter tables for improved readability.

Code: https://github.com/AlifAshrafee/CL-Under-Concept-Drift

Assigned Action Editor: ~changjian_shui1

Submission Number: 5508