Prompt Learning via Meta-Regularization

Jinyoung Park, Juyeon Ko, Hyunwoo J. Kim

Abstract: Pre-trained vision-language models have shown impres-sive success on various computer vision tasks with their zero-shot generalizability. Recently, prompt learning approaches have been explored to efficiently and effectively adapt the vision-language models to a variety of down-stream tasks. However, most existing prompt learning meth-ods suffer from task overfitting since the general knowl-edge of the pre-trained vision language models is forgot-ten while the prompts are finetuned on a small data set from a specific target task. To address this issue, we pro-pose a Prompt Meta-Regularization (ProMetaR) to im-prove the generalizability of prompt learning for vision-language models. Specifically, ProMetaR meta-learns both the regularizer and the soft prompts to harness the task-specific knowledge from the downstream tasks and task-agnostic general knowledge from the vision-language mod-els. Further, ProMetaR augments the task to gener-ate multiple virtual tasks to alleviate the meta-overfitting. In addition, we provide the analysis to comprehend how ProMetaR improves the generalizability of prompt tuning in the perspective of the gradient alignment. Our exten-sive experiments demonstrate that our ProMetaR improves the generalizability of conventional prompt learning meth-ods under base-to-base/base-to-new and domain general-ization settings. The code of ProMetaR is available at https://github.com/mlvlab/ProMetaR.