Visual Prompt Learning for Chinese Handwriting Recognition

Gang Yao, Ning Ding, Tianqi Zhao, Kemeng Zhao, Pei Tang, Yao Tao, Liangrui Peng

Abstract: Recognizing Chinese handwriting in unconstrained scenarios remains a challenging task due to wide variations in writing styles and imaging conditions. Recently, prompt learning in natural language processing has shown success in leveraging context awareness in various domains. This paper proposes to incorporate visual prompt learning into an encoder-decoder model for handwriting recognition. For the encoder, multi-scale meta prompts are incorporated to utilize contextual information in internal feature representations. For the decoder, additional character-level visual prompts are used along with the embeddings of previously predicted text to guide the decoding process. Experiments conducted on the SCUT-HCCDoc, SCUT-EPT and CASIA-HWDB Chinese handwriting datasets validate the effectiveness of the proposed methods.