Adaptive Multi-Modality Prompt Learning

Zongqian Wu, Yujing Liu, Mengmeng Zhan, Ping Hu, Xiaofeng Zhu

Abstract:

Although current prompt learning methods have successfully been designed to effectively reuse the large pre-trained models without fine-tuning their large number of parameters, they still have limitations to be addressed, i.e., without considering the adverse impact of meaningless patches in every image and without simultaneously considering in-sample generalization and out-of-sample generalization. In
this paper, we propose an adaptive multi-modality prompt learning to address the above issues. To do this, we employ previous text prompt learning and propose a new image prompt learning. The image prompt learning achieves in-sample and out-of-sample generalization, by first masking meaningless patches and then padding them with the learnable parameters and the information from texts. Moreover, each of the prompts provides auxiliary information to each other, further strengthening these two kinds of generalization. Experimental results on real datasets demonstrate that our method outperforms SOTA methods, in terms of different downstream tasks.

Primary Subject Area: [Content] Vision and Language

Secondary Subject Area: [Content] Multimodal Fusion

Relevance To Conference: This work introduces an efficient adaptive multimodal prompting method designed to enhance the adaptability of vision-language models for downstream tasks. Specifically, the proposed multimodal prompt learning effectively addresses both in-sample and out-of-sample generalization tasks concurrently. This contribution represents a pivotal step towards unlocking the full potential of multimodal data fusion.

Submission Number: 4255