InstructBrush: Learning Attention-based Visual Instruction for Image Editing

Ruoyu Zhao, Qingnan Fan, Fei Kou, shuai qin, Hong Gu, Wei WU, Pengcheng Xu, Mingrui Zhu, Nannan Wang, Xinbo Gao

Keywords: Image Editing.+Visual In-Context Learning.+Diffusion Models

TL;DR: Our method extracts editing effects from image pairs for editing tasks that are difficult for users to describe. It introduces a new instruction optimization and initialization method, achieving better instruction optimization and generalization.

Abstract: Diffusion-based image editing methods have garnered significant attention in image editing. However, despite encompassing a wide range of editing priors, these methods are helpless when handling editing tasks that are challenging for users to accurately describe. We propose InstructBrush, an inversion method for instruction-based image editing methods to bridge this gap. It extracts editing effects from example image pairs as editing instructions to guide the editing of new images. Two key techniques are introduced into InstructBrush, Attention-based Instruction Optimization and Transformation-oriented Instruction Initialization, to address the limitations of the previous method in terms of inversion effects and instruction generalization. To explore the ability of visual prompt editing methods to guide image editing in open scenarios, we establish a Transformation-Oriented Paired Benchmark (TOP-Bench). Quantitatively and qualitatively, our approach achieves superior performance in editing and is more semantically consistent with the target editing effects. The code and benchmark will be released upon acceptance.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3356