LAMP: Long-tailed Multimodal Prompt Learning for Vision-Language Models

Xucong Wang, Pengkun Wang, Zhe Zhao, Liheng Yu, Yang Wang

Keywords: Prompt Learning, Long-tailed Learning

TL;DR: Long-tailed Multimodal Prompt Tuning

Abstract: Prompt Learning (PL) has sparked growing interest as an efficient method for adapting large Vision-Language Models (VLMs) to downstream tasks. While most existing PL methods are designed for (nearly) balanced data, real-world datasets always exhibit a long-tailed distribution, which necessitates the design of PL methods specifically for imbalanced scenarios. This paper introduces our LAMP framework to enable VLMs better to learn from the long-tailed base classes and achieve non-biased predictions for both base and new classes. LAMP is integrated into specific intermediate layers of the frozen VLM, where for each layer we introduce three co-designed mechanisms: Multimodal Prompt Pool (MPP), Modality-Shared Prompts (MSP) and Load Balancing Optimization (LBO). MPP is motivated by the goal of boosting feature clustering to foster the mutual learning of head and tail classes, in which we introduce separated multimodal prompts that are dynamically combined into the model via similarity metrics. In addition to the prompts retrieved by MPP, we introduce globally shared MSPs to better adapt to cross-modal semantics and enhance the training robustness. Furthermore, to promote tight and discernible feature clustering via prompts, we treat each prompt as an expert and adopt a load-balancing technique from mixture-of-experts in LLMs, named LBO. LBO dynamically adjusts attention weights with an externally optimized bias, thereby making the activation of each prompt more evenly distributed and preventing the overfitting of head classes. Extensive experiments under various long-tailed settings demonstrate that our LAMP consistently outperforms other state-of-the-art methods.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 5979