SPARC: Continual learning beyond experience rehearsal and model surrogates

Prashant Shivaram Bhat, Laurens Niesten, Elahe Arani, Bahram Zonooz

Keywords: Continual learning, lifelong learning, computer vision, experience rehearsal, parameter isolation

Abstract:

Continual learning (CL) has become increasingly important as deep neural networks
(DNNs) are required to adapt to the continuous influx of data without retraining
from scratch. However, a significant challenge in CL is catastrophic forgetting (CF),
where learning new tasks erases previously acquired knowledge, either partially
or completely. Existing solutions often rely on experience rehearsal or full model
surrogates to mitigate CF. While effective, these approaches introduce substantial
memory and computational overhead, limiting their scalability and applicability in
real-world scenarios. To address this, we propose SPARC, a scalable CL approach
that eliminates the need for experience rehearsal and full-model surrogates. By
effectively combining task-specific working memories and task-agnostic semantic
memory for cross-task knowledge consolidation, SPARC results in a remarkable
parameter efficiency, using only 6% of the parameters required by full-model
surrogates. Despite its lightweight design, SPARC achieves superior performance
on Seq-TinyImageNet and matches rehearsal-based methods on various CL benchmarks. Additionally, weight re-normalization in the classification layer mitigates
task-specific biases, establishing SPARC as a practical and scalable solution for
CL under stringent efficiency constraints.

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5947