A Hybrid Corpus based Fine-grained Semantic Alignment Method for Pre-trained Language Model of Ancient Chinese Poetry

Jie Hong, Tingting He, Jie Mei, Ming Dong, Zheming Zhang, Xinhui Tu

Abstract: Ancient Chinese poetry (ACP) is a vital component of Chinese traditional culture. Enhancing the performance of related downstream tasks demands the development of high-quality pre-trained language models (PLMs) dedicated to ACP. Notably, the semantics of ACP significantly differ from modern Chinese. Existing PLMs have limited knowledge of ACP and are inadequately aligned with the semantic space of modern Chinese, which constrains the utility for tasks related to ACP. In this paper, we propose a fine-tuning strategy to establish a precise alignment between ACP and modern Chinese semantics on sentence level. This strategy involves the inclusion of corresponding modern Chinese translations alongside original ancient poems, creating a hybrid corpus. This corpus facilitates a more effective transfer of knowledge from existing PLMs to the domain of ACP. Furthermore, we employ a training strategy based on a glyph-based foundational PLM, enabling meticulous fine-tuning. Consequently, we develop a specialized PLM named CP-ChineseBERT. To evaluate the effectiveness of our proposed strategies, we conducted experiments on two real-world datasets, focusing on tasks related to ACP sentiment classification and ACP title prediction. The experimental results demonstrate the significant improvements in performance achieved through our innovative approaches.