BERT-KAIM: A Knowledge-Augmented Model for Domain-Specific Tasks with Learnable Activation Functions and Mixed Precision Training

Haiyang Jiang, Yingying Liu, Haixiang Wang, Yao Zheng, Kun Hou, Jingyuan Li, Haoliang Zhang

Abstract: The rapid development of pre-trained large language models, such as BERT, brings convenience but also comes with optimization challenges. The performance of large language models in domain-specific tasks has room for improvement. When dealing with tasks in specific domains, it may be beneficial to incorporate relevant domain knowledge for reasoning and training the model, allowing it to better adapt to domain-specific tasks and thereby improve its performance. However, an issue that arises from integrating too much knowledge is the potential for the original meaning to be distorted, leading to a decline in performance. By using graph activation functions and mixed precision training, the model can achieve better training speed and greater diversity and generalization when learning graph structures. Based on these challenges and improvements, we propose a new model, BERT-KAIM, which integrates domain knowledge graphs into BERT, incorporates learnable activation functions, and uses mixed precision training to enhance the model's performance, as validated by experimental results.