Improving Image Editing Models with Generative Data Refinement

Frederic Boesel, Robin Rombach

Keywords: image editing, diffusion models, synthetic datasets

TL;DR: We demonstrate that a refinement of the InstructPix2Pix dataset with the help of SDXL shows consistent improvements in downstream models; with SDXL as a base model we observe competitive performance compared to much more cost intensive methods.

Abstract:

Instruction-based generative image editing models allow an image to be modified based on a text prompt and have the potential to significantly improve the accessibility of image processing software. Like other generative models, they are highly dependent on the quality of their training dataset, and generating good editing datasets is an expensive task. In this paper, we show that a simple refinement
of the original InstructPix2Pix (Brooks et al., 2023) dataset using SDXL (Podell et al., 2023) leads to consistent improvements in downstream models. We finetune SDXL on our refined dataset and observe competitive performance to much more cost-intensive methods. We will make the dataset and models publicly available.

Submission Number: 74