Continual Learning by Reuse, New, Adapt and Skip: A Hierarchical Exploration-Exploitation Approach

ICLR 2026 Conference Submission18355 Authors

Keywords: Continual Learning, Adaptive Models

Abstract:

To effectively manage the complexities of real-world dynamic environments, continual learning must incrementally acquire, update, and accumulate knowledge from a stream of tasks—without suffering from catastrophic forgetting of prior knowledge. While this capability is innate to human cognition, it remains a significant challenge for modern deep learning systems.
At the heart of this challenge lies *the stability-plasticity dilemma*: the need to balance leveraging prior knowledge, integrating novel information, and allocating model capacity adaptively based on task complexity. In this paper, we propose a novel exemplar-free class-incremental continual learning (ExfCCL) framework that addresses these issues through a Hierarchical Exploration-Exploitation (HEE) approach. Our method centers on two key subsystems: (i) a HEE-guided neural architecture search (HEE-NAS) that enables a learning-to-adapt backbone via four primitive operations—reuse, new, adapt, and skip—thereby serving as an internal memory that dynamically updates selected components across tasks; and (ii) a task ID inference mechanism, which utilizes an external memory of task centroids to select the appropriate task-specific backbone and classifier during testing. We term our overall framework **CHEEM** (Continual Hierarchical-Exploration-Exploitation Memory). CHEEM is evaluated on the challenging MTIL and Visual Domain Decathlon (VDD) benchmarks using both Tiny and Base Vision Transformers. It significantly outperforms state-of-the-art prompting-based continual learning methods, closely approaching full fine-tuning upper bounds. Furthermore, it learns adaptive model structures tailored to individual tasks in a semantically meaningful way, demonstrating its effectiveness in exemplar-free continual learning scenarios.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 18355