Diffusion Meta-Prompts for Foundation Models

ICLR 2026 Conference Submission12960 Authors

Keywords: Diffusion, Meta-learning, Prompt learning, Prompt tuning, few-shot classification

TL;DR: We propose Diffusion Meta-Prompts (DMP), a diffusion-based method that generates prompts for foundation models with better memory, runtime efficiency and generalization across domains, tasks, and concepts.

Abstract: Parameter-efficient fine-tuning techniques, such as prompting, are now popular to adapt foundation models to many tasks. In this paper, we introduce a diffusion-based approach to model the distribution of learned foundation models prompts. Specifically, we propose a $\textbf{Diffusion Meta-Prompt (DMP)}$ model that generates prompts conditioned on text or prompt embeddings, and can be used to prompt both vision-language models and diffusion models for image synthesis. DMPs have several advantages: improved generalization of learned prompts; memory and runtime efficiency by eliminating the need to store and search over large repositories of prompts or LoRA weights; multiple applications ranging from open-set classification, to personalization or attribute control of image synthesis; support for operations like subject and concept composition, novel subject generation, negative prompting, and editing without explicit training. For open-set classification, DMP improves base-to-new class generalization, achieving upto $\textbf{3}$% average gain across 11 datasets with gains as high as $\textbf{7.8/5.4}$% on specific datasets such as Eurosat/UCF101 respectively. DMP also enhances domain, cross-dataset and cross-task generalization with $\sim$$\textbf{6-12}$% improvement for hierarchical classification task. For image synthesis tasks, DMP improves generalization and prompt compliance by $\textbf{1.4}$ points as measured by CLIP score and reduces storage requirements by $\textbf{91}$% while improving runtime efficiency by $\textbf{92}$% over retrieval methods.

Primary Area: generative models

Submission Number: 12960