Local Attention Augmentation for Chinese Spelling Correction

Shuo Wang, Chaodong Tong, Kun Peng, Lei Jiang

Abstract: Chinese spelling correction (CSC) is an important task in the field of natural language processing (NLP). While existing state-of-the-art methods primarily leverage pre-trained language models and incorporate external knowledge sources such as confusion sets, they often fall short in fully leveraging local information that surrounds erroneous words. In our research, we aim to bridge a crucial gap by introducing a novel CSC model that is enhanced with a Gaussian attention mechanism. This integration allows the model to adeptly grasp and utilize both contextual and local information. The model incorporates a Gaussian attention mechanism, which results in attention weights around erroneous words following a Gaussian distribution. This enables the model to place more emphasis on the information from neighboring words. Additionally, the attention weights are dynamically adjusted using learnable hyperparameters, allowing the model to adaptively allocate attention to different parts of the input sequence. In the end, we adopt a homophonic substitution masking strategy and fine-tune the BERT model on a large-scale CSC corpus. Experimental results show that our proposed method achieve a new state-of-the-art performance on the SIGHAN benchmarks.