Componential Prompt-Knowledge Alignment for Domain Incremental Learning

Kunlun Xu, Xu Zou, Gang Hua, Jiahuan Zhou

Abstract:

Domain Incremental Learning (DIL) aims to learn from non-stationary data streams across domains while retaining and utilizing past knowledge. Although prompt-based methods effectively store multi-domain knowledge in prompt parameters and obtain advanced performance through cross-domain prompt fusion, we reveal an intrinsic limitation: component-wise misalignment between domain-specific prompts leads to conflicting knowledge integration and degraded predictions. This arises from the random positioning of knowledge components within prompts, where irrelevant component fusion introduces interference.
To address this, we propose Componential Prompt-Knowledge Alignment (KA-Prompt), a novel prompt-based DIL method that introduces component-aware prompt-knowledge alignment during training, significantly improving both the learning and inference capacity of the model. KA-Prompt operates in two phases: (1) Initial Componential Structure Configuring, where a set of old prompts containing knowledge relevant to the new domain are mined via greedy search, which is then exploited to initialize new prompts to achieve reusable knowledge transfer and establish intrinsic alignment between new and old prompts. (2) Online Alignment Preservation, which dynamically identifies the target old prompts and applies adaptive componential consistency constraints as new prompts evolve. Extensive experiments on DIL benchmarks demonstrate the effectiveness of our KA-Prompt.
Our source code is available at https://github.com/zhoujiahuan1991/ICML2025-KA-Prompt.

Lay Summary:

Domain Incremental Learning (DIL) aims to adapt to continuous data streams with substantial domain shifts caused by variations in style, data quality degradation, and environmental changes. Although prompt-based methods effectively store multi-domain knowledge in prompt parameters and obtain advanced performance through cross-domain prompt fusion, we reveal an intrinsic limitation: component-wise misalignment between domain-specific prompts leads to conflicting knowledge integration and degraded predictions. This arises from the random positioning of knowledge components within prompts, where irrelevant component fusion introduces interference.
To address this, we propose Componential Prompt-Knowledge Alignment (KA-Prompt), a novel prompt-based DIL method that introduces component-aware prompt-knowledge alignment during training, significantly improving both the learning and inference capacity of the model. KA-Prompt can be applied to a variety of practical scenarios and tasks, including the continuously evolving AI agents, the multi-domain downstream adaptation of large vision models, and the lifelong environment adaptation of autonomous driving systems.

Link To Code: https://github.com/zhoujiahuan1991/ICML2025-KA-Prompt

Primary Area: Applications->Computer Vision

Keywords: Domain Incremental learning; Image Classification

Flagged For Ethics Review: true

Submission Number: 31