Style Expansion Without Forgetting for Handwritten Character Recognition

Jie Ruan, Zhenyu Weng, Jian Zhang, Yuqing Wang, Longhui Yu, Qiankun Gao, Yuesheng Zhu

Abstract: Handwritten character recognition (HCR) is still a challenging task due to diverse writing styles. In existing works, the recognition models for recognizing handwritten characters are usually trained with limited handwriting styles. However, there are always some new styles that are not included in the training sets in practical applications, degrading the recognition performance. Re-training the models with updated training sets containing new styles would increase the computational cost and complexity. In this paper, a new Style Expansion Learning HCR (StyleEL-HCR) problem is formulated to characterize this issue and a novel Reliable Prototype Augmentation (RePA) framework is developed for StyleEL-HCR. The RePA is composed of Soft Knowledge Distillation with replay (SKD), Character Prototype Augmentation (CPA), and Strict Gate Mechanism (SGM). SKD memorizes knowledge from old styles through distillation and replay, CPA learns representative information by memorizing character-representative prototypes and augmenting them in new learning phases to better distinguish different characters when the replay data is limited, and SGM augments the prototypes in a reliable way to improves the reliability of the recognition model. Adopting distillation and replay together with reliable prototype augmentation, RePA has the ability to learn new styles as well as preserve knowledge of old styles. Our experimental results have shown that the RePA can obtain better recognition performance for handwritten English, Chinese and digit characters compared to other methods, particularly when the handwritten Chinese character categories scale is large.