Continual Learning via Learning a Continual Memory in Vision Transformer

Chinmay Savadikar, Michelle Dai, Tianfu Wu

Keywords: Lifelong Learning, Continual Learning, Vision Transformers

TL;DR: We identify a component in Vision Transformers to serve as a task memory, and propose a sampling method leveraging task synergies for structurally updating the component with four basic operations (reuse, adapt, new and skip) for continual learning

Abstract: This paper explores continual learning (CL) using Vision Transformer (ViT) in streaming tasks under the challenging exemplar-free class-incremental (ExfCCL) setting. We formulate ExfCCL as a learning problem consisting of two key sub-systems: (i) task ID inference for test data, which selects appropriate task-specific head classifiers to accounting for varying class distributions across tasks and streams, and (ii) a dynamic learning-to-grow feature backbone that balances stability and plasticity, mitigating catastrophic forgetting through task synergies. Following the common protocol that the first task can train a ViT sufficiently well as the base model, we address these sub-systems from a continual memory learning perspective. To support task ID inference, we utilize an external memory mechanism that maintains task centroids computed by the base ViT throughout CL. For the feature backbone, we identify optimal placements for internal (parameter) memory to enable a dynamic, task-synergy guided growing feature backbone. We propose a Hierarchical Exploration-Exploitation (HEE) sampling-based neural architecture search (NAS) method that effectively learns task synergies by continually and structurally updating internal memory with four basic operations: _reuse_, _adapt_, _new_, and _skip_. Our approach, dubbed **Continual Hierarchical-Exploration-Exploitation Memory (CHEEM)**, is evaluated on the challenging Visual Domain Decathlon (VDD) and ImageNet-R benchmarks, demonstrating its effectiveness.

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8370