Multiple Local Prompts Distillation for Domain Generalization

Huaihai Lyu, Hantao Yao, Changsheng Xu

Abstract: Prompt tuning has been proven effective for Domain Generalization (DG) by enhancing the generalization capability of visual-language models with fewer learnable tokens. Existing methods adopt mostly inferring global-level individual prompts for the whole dataset to capture domain-invariant knowledge across different domains. However, since domain shifts exist, a single global-level individual prompt is easily overfitted to source domain datasets, thus lacking generalizability to the whole dataset's feature distribution. Moreover, fluctuations in the generalization performance during the training process in DG problems often pose significant challenges to model selection strategies. To address the aforementioned problems, inspired by the Mixture-of-Expert (MOE) and knowledge distillation, we propose a novel Multiple Local Prompts Distillation (MLPD) method to inject the knowledge of multiple local prompts into a unique global prompt, improving both the generalization and discriminative ability. To ensure the diversity of local prompts, we split the whole dataset into several subsets to infer the discriminative local prompts for each subset, which is further applied to generate the generability global prompt. Formally, for each subset, Meta Prompt Tuning (MPT) is proposed to constrain each local prompt to capture both the domain-specific and domain-shared generalization knowledge on the basis of the domain label and meta-learning mechanism. After that, Prompt Knowledge Distillation (PKD) is proposed to distill the knowledge captured in the local-level prompts into the global-level prompt with prompt-level and feature-level knowledge distillations. The final evaluation on multiple benchmarks underscores the effectiveness of the proposed MLPD, e.g., achieving mAPs of 97.3%, 84.8%, 85.2%, 57.3%, and 60.7% on PACS, VLCS, OfficeHome, TerraIncognita, and DomainNet, respectively.

External IDs:doi:10.1109/tmm.2025.3607719