Prototypical evoluation for few-shot learning in vision-language model adaptation
Keywords: CLIP, few-shot classification
Abstract: Vision-Language Models (e.g., CLIP), with their immense capacity and extensive exposure to vast data during pre-training, have demonstrated a strong ability to capture real-world concepts. When fast adapted to downstream tasks with only a few labeled samples, parameter-efficient methods, such as prompt-based and adapter-based approaches, which adjust only a small portion of the parameters, have proven effective in reducing the escalating costs in large vision-language models. However, conventional efficient fine-tuning techniques, using task-specific objectives like cross-entropy loss, often lead to overfitting the downstream data distributions. This overfitting diminishes the model’s ability to retain its original generalization capacity, especially on out-of-distribution (OOD) samples. Unlike the pretraining stage, where rich textual descriptions are available, fine-tuning is typically constrained to using only class names. This creates suboptimal text-image alignment in the shared feature space, as it may exacerbate image feature variance within the same class. To address this issue, we propose Prototypical Evolutionary Adaptation (PEA), leveraging off-the-shelf image centroids as prototypes to regulate image feature variance, mitigating the excessive feature variance within the same class caused by selective bias. Additionally, we introduce learnable shift vectors to capture the dynamics of class prototypes, ensuring that they remain compact and informative. Experiments across diverse datasets and model architectures in few-shot learning demonstrate that our approach consistently outperforms existing methods while maintaining robust generalization under varying distribution shifts.
Primary Area: transfer learning, meta learning, and lifelong learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 10921
Loading