Self-Prompting Diffusion Transformer for Open-Vocabulary Scene Text Edit via In-Context Learning

Hongxi Li, Tong Wang, WU CHENGJING, Tianbao Liu, Jiangtao Yao, Xiaochao Qu, Xinxiao Wu, Luoqi Liu, Ting Liu

Abstract:

Scene text editing aims to modify text in a target region of an image while preserving its background style and texture.
Existing methods rely solely on image background information while neglecting the visual details of target regions,
which discards stylistic features in the original text and essentially degrades the task to text rendering.
Moreover, the conditions imposed by pre-trained glyph encoder limit the scope of editable text.
To address these issues, this paper proposes a self-prompting scene text editing method, which constructs style and glyph prompts directly from the original image without additional style or glyph encoders.
We employ a two-stage training strategy, where the diffusion transformer is first trained on large-scale self-supervised datasets and subsequently refined with a small set of paired images.
By leveraging the in-context learning capability of FLUX-Fill, it achieves open-vocabulary and style-consistent text editing.
Experimental results on various languages demonstrate that our method achieves the state-of-the-art performance in both text accuracy and style consistency.

Lay Summary:

Scene text editing aims to modify text inside images while keeping the edited results natural and visually consistent. However, existing methods often fail to preserve the original text’s style and are usually limited to a fixed set of words or languages.

We propose a self-prompting text editing method that learns directly from the original image, without requiring additional text encoders. By leveraging the contextual learning ability of modern generative models, our method can edit previously unseen text while preserving the original visual style.

Our approach supports open-vocabulary multilingual editing across languages such as Chinese, English, Japanese, Korean, Russian, and Thai. Experiments show that it produces more accurate and realistic editing results than existing methods.

Primary Area: Applications->Computer Vision

Keywords: Scene text edit, Image edit, Diffusion transformer

Submission Number: 33685