Weighted Multi-Prompt Learning with Description-free Large Language Model Distillation

Sua Lee, Kyubum Shin, Jung Ho Park

Keywords: Prompt learning, Vision-language models, Large language models, Few-shot image recognition

TL;DR: A new approach to distill the LLM information into VLM prompts without descriptions, while also weighting important prompts in a multi-prompt setting.

Abstract:

Recent advances in pre-trained Vision Language Models (VLM) have shown promising potential for effectively adapting to downstream tasks through _prompt learning_, without the need for additional annotated paired datasets.
To supplement the text information in VLM trained on correlations with vision data, new approaches leveraging Large Language Models (LLM) in prompts have been proposed, enhancing robustness to unseen and diverse data.
Existing methods typically extract text-based responses (i.e., _descriptions_) from LLM to incorporate into prompts; however, this approach suffers from high variability and low reliability.
In this work, we propose **De**scription-free **Mul**ti-prompt Learning(**DeMul**), a novel method that eliminates the process of extracting descriptions and instead directly distills knowledge from LLM into prompts.
By adopting a description-free approach, prompts can encapsulate richer semantics while still being represented as continuous vectors for optimization, thereby eliminating the need for discrete pre-defined templates.
Additionally, in a multi-prompt setting, we empirically demonstrate the potential of prompt weighting in reflecting the importance of different prompts during training.
Experimental results show that our approach achieves superior performance across 11 recognition datasets.

Primary Area: foundation or frontier models, including LLMs

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8536