Variational Inference based Probabilistic Prompt for Rehearsal-Free Continual Learning

Xin Du, Sikun Yang, Wouter Duivesteijn, Mykola Pechenizkiy

Keywords: Variational Inference, Prompt, Continual Learning, Vision Transformers

Abstract: Continual learning aims to enable models to learn a sequence of tasks without catastrophic forgetting, a phenomenon where new information overwrites previously acquired knowledge. Traditional solutions for this problem, including regularization, replay buffers, and dynamic architectures, struggle with trade-offs in scalability, privacy, and adaptability. Prompt-based learning, initially developed in NLP, offers parameter-efficient alternatives by prepending learnable vectors to input tokens. However, existing prompt methods in continual learning, such as L2P and DualPrompt, rely on deterministic selection mechanisms that lack uncertainty modeling, making them less effective in dynamic and ambiguous task scenarios. In this work, we propose a novel framework called Variational Inference based Probabilistic Prompt (VPrompt) that introduces a stochastic latent variable formulation over prompt selection using variational inference. Our method learns an approximate posterior distribution over prompt assignments conditioned on inputs, and regularizes this with a uniform prior to ensure diversity and mitigate overconfidence. This probabilistic mechanism enables uncertainty-aware adaptation, improves robustness under domain shift, and eliminates the need for task labels or rehearsal buffers. We evaluate our method across Split CIFAR100, Split ImageNet-R, and a diverse 5-dataset benchmark. VPrompt consistently outperforms state-of-the-art baselines, including CODA-Prompt, L2P, DualPrompt, regularized and rehearsal-based methods, in terms of average accuracy and reduced forgetting. These results confirm that modeling uncertainty at the prompt level offers a scalable, buffer-free, and more flexible solution for continual learning.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 7276