SOPRANO: Synergistic Optimization with Progressive Replay and Adaptive Network Orchestration for Continual Learning

ICLR 2026 Conference Submission16929 Authors

Keywords: continual learning, class-incremental learning, experience replay, knowledge distillation

Abstract: Continual learning remains a core challenge for deep neural networks, where models catastrophically forget prior knowledge when trained on new tasks. We introduce \soprano{} (\textbf{S}ynergistic \textbf{O}ptimization with \textbf{P}rogressive \textbf{R}eplay and \textbf{A}daptive \textbf{N}etwork \textbf{O}rchestration), a framework that combines balanced memory replay and adaptive knowledge distillation with task-aware optimization. Unlike approaches that rely on fixed replay schedules or rigid regularizers, \soprano{} adapts its learning dynamics to task characteristics. On CIFAR-100 (5/10/20 tasks) and CIFAR-10-5, \soprano{} delivers strong performance: \textbf{56.4$\pm$0.6}\% on CIFAR-100-5, \textbf{46.7$\pm$0.5}\% on CIFAR-100-10, \textbf{33.8$\pm$0.6}\% on CIFAR-100-20, and \textbf{58.5$\pm$1.1}\% on CIFAR-10-5. On CIFAR-100-5, this is about \textbf{3.3$\times$} the accuracy of strong replay baselines (DER: 17.2$\pm$0.3\%, DER++: 17.1$\pm$0.2\%) and far exceeds regularization-based methods (EWC: 10.8$\pm$7.0\%). \soprano{} also achieves markedly lower forgetting (e.g., \textbf{7.6$\pm$0.2}\% vs.\ 79.1$\pm$0.4\% for DER and 68.7$\pm$1.2\% for EWC on CIFAR-100-5). Ablation studies confirm complementary contributions from balanced replay and distillation. Code will be released upon acceptance.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 16929