Synchronous Scene Text Spotting and Translating

Qiuyue Ma, Yaping Zhang, Yu Zhou

Keywords: multimodal machine translation

TL;DR: a novel scene text understanding and translating framework keep layout information

Abstract:

Text image machine translation aims to translate the content of textual regions in images from a source language to a target language. Compared with traditional document, images captured in natural scenes have more diverse text and more complex layout, posing challenges in recognizing text content and predicting reading order within each text region. Current methods mainly adopt pipeline pattern, in which models for text spotting and translating are trained separately. In this pattern, translation performance is affected by propagation of mispredicted reading order and text recognition errors. In this paper, we propose a scene text image machine translation approach by implementation of synchronous text spotting and translating. A bridge and fusion module is introduced to make better use of multi-modal feature. Besides, we create datasets for both Chinese-to-English and English-to-Chinese image translation. Experimental results substantiate that our method achieves state-of-the-art translation performance in scene text field, proving the effectiveness of joint learning and multi-modal feature fusion.

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1700