GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Models

ICLR 2025 Conference Submission473 Authors

Keywords: llms, vlms, prompt optimization

TL;DR: GLOV uses LLMs as implicit optimizers for VLMs, generating and refining task-specific prompts, leading to up to 57.5% improved performance on diverse vision tasks.

Abstract:

In this work, we propose a novel method (GLOV) enabling Large Language Models (LLMs) to act as implicit Optimizers for Vision-Langugage Models (VLMs) to enhance downstream vision tasks.
Our GLOV meta-prompts an LLM with the downstream task description, querying it for suitable VLM prompts (e.g., for zero-shot classification with CLIP).
These prompts are ranked according to a purity measure obtained through a fitness function.
In each respective optimization step, the ranked prompts are fed as in-context examples (with their accuracies) to equip the LLM with the knowledge of the type of text prompts preferred by the downstream VLM.
Furthermore, we also explicitly steer the LLM generation process in each optimization step
by specifically adding an offset difference vector of the embeddings from the \textit{positive} and \textit{negative} solutions found by the LLM, in previous optimization steps, to the intermediate layer of the network for the next generation step.
This offset vector steers the LLM generation toward the type of language preferred by the downstream VLM, resulting in enhanced performance on the downstream vision tasks.
We comprehensively evaluate our GLOV on 16 diverse datasets using two families of VLMs, i.e., dual-encoder (e.g., CLIP) and encoder-decoder (e.g., LLaVa) models --
showing that the discovered solutions can enhance the recognition performance by up to $15.0$% and $57.5$% ($3.8$% and $21.6$% on average) for these models.

Primary Area: foundation or frontier models, including LLMs

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 473