TIPS: Two-Level Prompt for Rehearsal-free Continual Learning
Keywords: Continual Learning; Prompt Learning; Catastrophic Forgetting
TL;DR: This work introduces a two-level prompt selection strategy to enhance stability and a semantic distillation module to improve plasticity, achieving a best balance compare SOTA.
Abstract: Continual learning based on prompt tuning creates a key-value pool, where these key-value pairs are called prompts. Prompts are retrieved using input images as queries and input into a frozen backbone network. It requires training only a few parameters to quickly adapt to downstream tasks. Compared to other traditional Continual learning methods, it is more effective in resisting catastrophic forgetting. However, the effectiveness of these methods heavily depends on the selection strategy.
Most existing methods overlook the model plasticity since they focus on solving the model's stability issues, leading to a sharp decline in performance for new tasks in long task sequences of incremental learning.
To address these limitations, we propose a novel prompt-based continual learning method called TIPS, which mainly consists of two modules: (1) design a novel two-level prompt selection strategy combined with a set of adaptive weights for sparse joint tuning, aiming to improve the accuracy of prompt selection; (2) design a semantic distillation module that enhances the generalization ability to unknown new classes by creating a language token and utilizing the encapsulated semantic information of class names.
We validated TIPS on four datasets across three incremental scenarios.
Our method outperformed the current state of the art (SOTA) by 2.03%, 4.78%, 1.18%, and 5.59% on CIFAR (10 tasks), ImageNet-R (20 tasks), CUB (10 tasks), and DomainNet (20 tasks).
Notably, our approach consistently surpasses or matches SOTA in all settings, maintaining stable prompt selection accuracy throughout multiple incremental learning sessions.
Supplementary Material: zip
Primary Area: transfer learning, meta learning, and lifelong learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 6588
Loading