CoSPA: An Improved Masked Language Model for Chinese Spelling Correction

Shoujian Yang, Lian Yu

Keywords: Chinese Spelling Correction

Abstract: Existing BERT-based models for Chinese spelling correction (CSC) have three issues. 1) Bert tends to correct a correct low-frequency collocation into a high-frequency and leads to over-correcting. 2) The current learned knowledge for CSC ignores the phonic and glyph aspects of each character and unable to differentiate a near-phonic or a near-visual conversion. 3) Two-dimensional glyph information of Chinese characters is overlooked and fails to discover near-visual misused characters. This paper proposes a hybrid approach, CoSPA, to address these issues. 1) This paper proposes an alterable copy mechanism to alleviate over-correcting by jointly learning to copy a correct character from input sentence, or generate a character from BERT. No method has used copy mechanism in BERT for CSC. 2) The attention mechanism is further applied on the phonic and shape representation of each character at the output layer. 3) Shape representation is enhanced by mining character glyph with ResNet, and fused with stroke representation via an adaptive gating unit. The experimental results show that CoSPA outperforms the previous state-of-the-art methods on SIGHAN2015 datasets.