DiffUTE: Universal Text Editing Diffusion Model

Haoxing Chen, Zhuoer Xu, Zhangxuan Gu, jun lan, 行 郑, Yaohui Li, Changhua Meng, Huijia Zhu, Weiqiang Wang

Keywords: Diffusion model, text editing, self-supervied learning

TL;DR: This paper propose a unified model to edit any texts in images.

Abstract: Diffusion model based language-guided image editing has achieved great success recently. However, existing state-of-the-art diffusion models struggle with rendering correct text and text style during generation. To tackle this problem, we propose a universal self-supervised text editing diffusion model (DiffUTE), which aims to replace or modify words in the source image with another one while maintaining its realistic appearance. Specifically, we build our model on a diffusion model and carefully modify the network structure to enable the model for drawing multilingual characters with the help of glyph and position information. Moreover, we design a self-supervised learning framework to leverage large amounts of web data to improve the representation ability of the model. Experimental results show that our method achieves an impressive performance and enables controllable editing on in-the-wild images with high fidelity. Our code will be avaliable in \url{https://github.com/chenhaoxing/DiffUTE}.

Submission Number: 1165