Advancing Prompt Learning through an External Layer

Fangming Cui, Xun Yang, Chao Wu, Liang Xiao, Xinmei Tian

Abstract: Prompt learning represents a promising method for adapting pre-trained vision-language models (VLMs) to various downstream tasks by learning a set of text embeddings. One challenge inherent to these methods is the poor generalization performance due to the invalidity of the learned text embeddings for unseen tasks. A straightforward approach to bridge this gap is to freeze the text embeddings in prompts, which results in a lack of capacity to adapt VLMs for downstream tasks. To address this dilemma, we propose a paradigm called EnPrompt with a novel External Layer (EnLa). Specifically, we propose a textual external layer and learnable visual embeddings for adapting VLMs to downstream tasks. The learnable external layer is built upon valid embeddings of pre-trained CLIP. This design considers the balance of learning capabilities between the two branches. To align the textual and visual features, we propose a novel two-pronged approach: i) we introduce the optimal transport as the discrepancy metric to align the vision and text modalities, and ii) we introduce a novel strengthening feature to enhance the interaction between these two modalities. Four representative experiments (i.e., base-to-novel generalization, few-shot learning, cross-dataset generalization, domain shifts generalization) across 15 datasets demonstrate that our method outperforms the existing prompt learning method.

Primary Subject Area: [Content] Multimodal Fusion

Secondary Subject Area: [Content] Vision and Language

Relevance To Conference: Prompt learning represents a promising method for adapting pre-trained visual-language models (VLMs) to various downstream tasks. Prompt learning plays a significant role in advancing multimedia/multimodal processing by enabling effective integration and understanding of multiple modalities, such as text, images. Prompt learning helps establish a meaningful semantic connection between different modalities. By aligning prompts with visual or textual features, prompt learning enables the model to understand and reason about the relationships and interactions between different modalities.

Submission Number: 1954