MagicRemover: Tuning-free Text-guided Image Inpainting with Diffusion Models

Siyuan Yang, Lu Zhang, Liqian Ma, Jingjing Fu, Yu LIU, You He

Keywords: Stable Diffusion, Inpainting, Tuning-free

Abstract:

Image inpainting aims to fill in the missing pixels with visually coherent and semantically plausible content. Despite the great progress brought from deep generative models, this task still suffers from \emph{i.} the difficulties in large-scale realistic data collection and costly model training; and \emph{ii.} the intrinsic limitations in the traditionally user-defined binary masks on objects with unclear boundaries or transparent texture.
%
In this paper, we propose \emph{\textbf{MagicRemover}}, a tuning-free method that leverages the powerful diffusion models for text-guided image inpainting. We introduce an attention guidance strategy to constrain the sampling process of diffusion models, enabling the erasing of instructed areas and the restoration of occluded content. We further propose a classifier optimization algorithm to facilitate the denoising stability within less sampling steps.
%
Extensive comparisons are conducted among our MagicRemover and state-of-the-art methods including quantitative evaluation and user study, demonstrating the significant improvement of MagicRemover on high-quality image inpainting.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3032