StyleGuide: Crafting visual style prompting with negative visual query guidance

Jaeseok Jeong, Junho Kim, Gayoung Lee, Yunjey Choi, Youngjung Uh

Keywords: Style transfer, Generative models, Diffusion models, Visual prompting, Visual instruction, Computer vision, Content creation, Image synthesis

TL;DR: We tackle visual style prompting, reflecting style elements from reference images and contents from text prompts, in a training-free manner.

Abstract: In the domain of text-to-image generation, diffusion models have emerged as powerful tools. Recently, studies on visual prompting, where images are used as prompts, have enabled more precise control over style and content. However, existing methods often suffer from content leakage, where undesired elements from the visual style prompt are transferred along with the intended style (content leakage). To address this issue, we 1) extends classifier-free guidance (CFG) to utilize swapping self-attention and propose 2)negative visual query guidance (NVQG) to reduce the transfer of unwanted contents. NVQG employ negative score by intentionally simulating content leakage scenarios which swaps queries instead of key and values of self-attention layers from visual style prompts. This simple yet effective method significantly reduces content leakage. Furthermore, we provide careful solutions for using a real image as a visual style prompts and for image-to-image (I2I) tasks. Through extensive evaluation across various styles and text prompts, our method demonstrates superiority over existing approaches, reflecting the style of the references and ensuring that resulting images match the text prompts.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5943